CAREER: Inferring Physics from Images: Inverse Rendering and Simulation with Generative Priors

This project aims to teach computers how to understand the physical properties of the world from images and videos. When humans look at a video, they intuitively understand how light reflects off surfaces and how objects move under force. However, artificial intelligence systems struggle to understand these physical rules. This project supports the development of new computer vision systems that can automatically figure out an object's three-dimensional shape, what it is made of, how it reacts to light, and how it moves, all from standard video recordings. Teaching computers this kind of physical reasoning will have major benefits for society. In healthcare, these tools can help robotic surgical systems safely navigate inside of the human body by understanding how tissues stretch and deform. In manufacturing, they can help robots handle delicate or flexible materials. The project also supports educational goals by creating new courses that teach physical understanding using artificial intelligence, providing research experiences for undergraduates, and attracting the participation of high school students and individuals from non-computing backgrounds to inspire the next generation of scientists and engineers. The technical goal of this project is to develop a generalizable machine perception framework that jointly infers three-dimensional object shape, parameters related to various physical properties of the object, and external physical entities from sparse visual inputs. This problem is technically challenging due to the ambiguity in estimating multiple spatial and physical properties from sparse-view videos. To resolve this ambiguity, the investigator will develop new techniques that combine physics-based modeling with strong statistical priors from generative models in an end-to-end framework. The investigator will achieve this through three main research activities. First, the project will develop generalizable inverse rendering models for static scenes by using controllable image or video diffusion models as priors to recover shape, reflectance, and lighting. Second, the project will tackle inverse simulation for dynamic scenes by combining feed-forward neural networks with a differentiable physics simulator to estimate initial physical states and material deformation parameters. Third, the project will unify these physical domains by building a controllable video generative model conditioned on both the physical parameters of lighting and deformation, enabling the joint estimation of object appearance and dynamics. The resulting algorithms will be validated on both synthetic and real-world datasets. Furthermore, the investigator will demonstrate the real-world impact of this foundational framework by applying it to healthcare challenges involving medical endoscopy videos. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2543161 | Program: 01003031DB NSF RESEARCH & RELATED ACTIVIT,01002930DB NSF RESEARCH & RELATED ACTIVIT,01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Roni Sengupta | Institution: University of North Carolina at Chapel Hill, CHAPEL HILL, NC | Award Amount: $328,476 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2543161 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2543161.html