CAREER: Self-Supervised Video Representation Learning for Machine Perception

Artificial intelligence systems can now generate realistic video, but they still struggle to learn the world knowledge needed to understand how environments change over time, anticipate the consequences of actions, and support decision- making in the physical world. This limitation is a major barrier to building machines that can safely and effectively assist people in homes, workplaces, and scientific settings. By developing learning methods that extract action-relevant structure directly from raw video and other sensor data, this project will help lay the foundation for more capable and adaptable intelligent systems, with potential benefits for robotics, scientific discovery, and other applications that require reliable machine perception. The project will also create open educational materials and mentorship activities that train students across vision, robotics, and machine learning. This project develops a self-supervised framework for video representation learning that separates efficient perception modules from generative world models, enabling the discovery of compact representations of scene state, motion, and action from raw sensory streams without dense human annotation. The research will study learning objectives and architectures that support long-context prediction, planning, and action-conditioned world modeling, while also yielding representations that can implicitly support conventional vision capabilities such as 3D reconstruction, motion estimation, and segmentation through a single scalable learning objective. By unifying perception, prediction, and planning in one framework, this research agenda aims to advance general machine perception for embodied intelligence. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2543631 | Program: 01002930DB NSF RESEARCH & RELATED ACTIVIT,01002627DB NSF RESEARCH & RELATED ACTIVIT,01003031DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Vincent Sitzmann | Institution: Massachusetts Institute of Technology, CAMBRIDGE, MA | Award Amount: $360,000 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2543631 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2543631.html