CRCNS Research Proposal: Prediction of semantic information across saccades

This project aims to investigate how the brain seamlessly stitches together the meaning of our surroundings even as our eyes constantly move. Humans and other primates actively sample the world by making rapid eye movements and briefly pausing to fixate on particular items. While scientists know that the brain connects simple visual details between these fixations, the neural mechanisms that link complex, high-level semantic information across multiple eye movements remain unknown. To investigate this, the project aims to record and analyze brain activity and eye movements from humans and non-human primates as they view natural images and videos. These data can guide the development of computational models of biological vision and inform the design of highly efficient future artificial intelligence (AI) systems. Additionally, the project aims to produce valuable open neural data and provide interdisciplinary training for future researchers. The primate visual system relies heavily on recurrent connections to process visual information, and it maintains its tuning without external supervision. The central hypothesis of the project is that predicting the semantic content of the next fixation can serve the role of self-supervised learning, yielding adaptive intelligent behavior, much like prediction is used to tune AI models. The project has two primary aims: first, to test alternative mechanisms of semantic prediction based on either current foveal vision or peripheral preview. Second, it aims to determine whether and how semantics are integrated recurrently across multiple eye movements. To achieve these goals, the investigators plan to analyze intracranial and scalp recordings alongside eye movements during free-viewing tasks. The methodological approach combines human experiments utilizing natural and controlled stimuli, cortical inactivation in non-human primates, and the development of computational models to evaluate prediction error signals under various hypotheses. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2605711 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Lucas Parra | Institution: CUNY City College, NEW YORK, NY | Award Amount: $1,638,742 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2605711 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2605711.html