CAREER: Video Understanding Like Humans Do: Empathy, Reasoning, and Privacy-Aware Learning

Understanding human behavior in videos goes beyond recognizing objects or actions. It involves interpreting emotion, reasoning about context, and respecting privacy. Current artificial intelligence systems often rely on simple visual patterns, struggle to explain their decisions, and are trained on data that may include sensitive personal information. These limitations make it difficult to use them in real-world settings such as accessibility. This project addresses these challenges by developing methods for human-aligned video understanding that can interpret emotional and social context, provide clear explanations, and operate on privacy-preserving data. The outcomes will support socially responsible artificial intelligence systems, improve accessibility tools such as audio descriptions for blind and low-vision audiences, and provide open resources that advance research, education, and public understanding of artificial intelligence. This project introduces a unified, closed-loop learning framework in which tasks, evaluation, and data are jointly optimized to improve model performance. The approach integrates three components. Empathy-driven modeling captures social and emotional context. Reasoning-aware evaluation diagnoses and improves intermediate model behavior. Privacy-preserving data generation enables the creation of semantically meaningful data without exposing sensitive information. Within this framework, benchmark datasets and evaluation protocols are used to identify model failures and guide iterative improvements to both models and data. The project will produce benchmark datasets, evaluation tools, and privacy-preserving generative models. It will also integrate these resources into education and outreach activities that broaden participation in artificial intelligence. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2541968 | Program: 01003031DB NSF RESEARCH & RELATED ACTIVIT,01002627DB NSF RESEARCH & RELATED ACTIVIT,01002930DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: SouYoung Jin | Institution: Dartmouth College, HANOVER, NH | Award Amount: $297,131 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2541968 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2541968.html