CAREER: Data-centric Capabilities for Vision Models

Modern computer vision has advanced rapidly in image understanding, editing, and generation. However, strong performance alone is insufficient for real-world deployment. Practical systems must also be safe, accountable, interpretable, and maintainable, especially as AI becomes deeply embedded in everyday life. Today’s vision systems are typically built on large, fully parametric models that encode vast amounts of training data in complex, non-intuitive ways. This makes it difficult to diagnose undesirable outputs, adapt models to shifting data distributions, or trace specific behaviors back to the data that influenced them. This project addresses these limitations by developing a new class of data-centric models, where model behavior can be more directly interpreted, attributed, and updated through explicit connections to training data. The goal is to enable more controllable and maintainable AI systems. Prior work has explored post-hoc approaches for data-centric capabilities; these methods are applied after a model is trained. While useful, these techniques do not address the underlying issue of fully parametric model design. In contrast, non-parametric methods such as k-nearest neighbors naturally provide strong data-centric properties, since predictions can be directly linked to training examples, though they often fall short in performance compared to modern deep models. This project proposes a semi-parametric paradigm that combines the strengths of both approaches: the performance of parametric models with the interpretability and controllability of non-parametric methods. The research is organized around four thrusts: developing semi-parametric architectures as a core framework and applying them to problems of unlearning, data attribution, and model customization. These models explicitly incorporate training data at inference time, enabling predictions to be traced back to specific examples. The project will further investigate how to ensure models effectively utilize this data at test-time, through architectural design (e.g., equivariance) and improved training strategies. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2540109 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT,01002930DB NSF RESEARCH & RELATED ACTIVIT,01003031DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Raymond Yeh | Institution: Purdue University, WEST LAFAYETTE, IN | Award Amount: $339,075 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2540109 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2540109.html