CAREER: Generative Augmented Reality

People increasingly rely on artificial intelligence to answer questions, generate content, and support learning. Yet most of these systems still live on flat screens, even when the task itself takes place in the physical world. That gap matters when someone is repairing equipment, learning hands-on skill, conducting a scientific experiment, working on a construction site, or creating something with physical materials. This project studies a new kind of augmented reality that can interpret a person’s physical surroundings, generate useful visual and interactive content for that situation, and place that content directly into the real physical environment. The goal is to make artificial intelligence more naturally present in everyday settings where people need guidance, feedback, or creative support in the moment. Potential uses include job training, design work, and science education. The project also contributes new educational materials in augmented reality and artificial intelligence, research training for undergraduate and graduate students, and outreach activities for younger learners. This project establishes a new concept called Generative Augmented Reality (Gen AR) as a framework for combining computer vision, large language models, and generative models within interactive augmented reality systems. The research develops methods for extracting context from the physical environment, generating content that fits the current task and setting, and embedding that content into the user’s view in ways that remain understandable and controllable. It also creates a general development platform so researchers, developers, and learners can build new Gen AR applications without starting from scratch. The project evaluates these systems in areas such as hands-on skills training, creative design, and science learning, and uses human-subject studies to examine user agency, trust, understanding, and privacy. The expected result is a set of technical methods, design tools, and human-centered guidelines for building augmented reality systems that use generative artificial intelligence responsibly and effectively. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2543251 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT,01003031DB NSF RESEARCH & RELATED ACTIVIT,01002930DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Ryo Suzuki | Institution: University of Colorado at Boulder, Boulder, CO | Award Amount: $388,110 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2543251 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2543251.html