CAREER: Situated Visual Augmentation for Human-AI Complementarity in Physical Spaces

Artificial intelligence (AI) is becoming a powerful tool for helping people make decisions, but most AI systems still deliver advice on desktop screens. This is a poor fit for people working in physical environments, such as surgeons, facility teams, first responders, and coaches. Augmented reality (AR) can place AI guidance directly into these settings, but simply moving information off a screen is not enough. In these settings, people must divide attention among the environment, movement, and the task itself, which can lead them to accept or reject AI advice too readily. This award develops adaptive AR interfaces that help people engage with AI more deliberately in real time while avoiding unnecessary disruption to the physical task. By improving how people and AI work together in real-world context, the project can support safer, more accurate, and more accountable decision-making in settings where errors are costly. The project will also train students in spatial computing, create open tools and learning materials, and broaden participation through courses, tutorials, and workshops. This award studies how adaptive situated visualizations in AR shape appropriate reliance on AI during decision-making tasks. The research will first conduct controlled experiments to identify which visual design choices, such as placement, visual prominence, lighting consistency, proximity, and scale, encourage overly fast judgments or more deliberate reasoning. It will then use these findings to design and evaluate adaptive visual prompts that change in response to user behavior, task demands, and AI confidence, for example by highlighting overlooked alternatives or revealing uncertainty at important moments. Finally, the project will develop learning-based methods that decide when and how to present these prompts using real-time signals from the user, the environment, and the AI system. The methods will be tested in sports analytics and facility management scenarios, producing general design principles and open-source tools for AR systems that help people and AI make better decisions together in physical spaces. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2543054 | Program: 01003031DB NSF RESEARCH & RELATED ACTIVIT,01002930DB NSF RESEARCH & RELATED ACTIVIT,01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Zhutian Chen | Institution: University of Minnesota-Twin Cities, MINNEAPOLIS, MN | Award Amount: $400,118 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2543054 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2543054.html