CAREER: Decoding the Cognitive-Motor Interface: How Visual Working Memory Transforms Sequential Plans into Coordinated Actions

Complex skills, from performing surgery to flying a jet, are built from long sequences of eye and hand movements that the brain must organize into a smooth, accurate performance. The eyes lead this process: long before a surgeon’s hands finish one movement, the eyes have already located what comes next. Whereas a novice’s gaze remains fixed on the hands, an expert’s eyes move ahead as the hands complete the current action. This fluent coordination frees attention for split-second decisions that depend on working memory, the brain’s limited short-term storage system, to hold the plan in mind while gaze shifts elsewhere. Because working memory can hold only a handful of items, skilled sequences remain fragile: a single distraction, such as an unexpected sound, can pull the gaze off course and disrupt the flow of movement. This project measures eye movements, hand movements, and brain activity as people learn complex skills. It explores how the brain builds and protects long, precise movement sequences and how it adapts to, or fails to adapt to, distractions that compete for attention. More broadly, this work advances understanding of how cognition, decision-making, and motor control interact in demanding, distraction-prone tasks in both everyday and expert settings. These insights can inform how such skills are taught and assessed, support rehabilitation after brain injury, and shape technologies that help people sustain the deep, unbroken focus that precise performance requires. They may also help explain why critical alerts, such as changes on a patient monitor, are sometimes missed when attention is fully absorbed by a demanding task. Although the limits of working memory are well established, how the brain uses such limited capacity to plan and execute long, coordinated movement sequences, such as those required in complex surgeries, remains poorly understood. This project tests the idea of hierarchical compression: the brain first groups individual visual items into larger memory units (chunking), and these chunks then allow neighboring movements to blend smoothly together (coarticulation). To study this link directly, the project develops a computational framework grounded in information theory to quantify the moment-to-moment relationship between gaze and hand movement. Participants perform sequences of reaching movements using a robotic arm while their eye movements and brain activity (EEG) are recorded, yielding measures of how attention is organized and how well gaze predicts the next action. The project examines 1) how working memory organizes the planning and execution of movement sequences; 2) how competing visual information disrupts the translation of memory into coordinated action as cognitive demands increase; and 3) how movement demands reshape attention and memory. By combining movement-based and information-theoretic measures with computational modeling and neural recordings, the project builds a unified account of how the brain turns limited memory into skilled, intelligent action. This award was co-funded by the Perception, Action and Cognition program and the Mind, Machine and Motor Nexus. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2544105 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Tarkeshwar Singh | Institution: Pennsylvania State Univ University Park, UNIVERSITY PARK, PA | Award Amount: $775,450 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2544105 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2544105.html