CRCNS Research Proposal: Generative models of neural activity driving conversational speech in longitudinal dialogues

Language is a unique expression of our human intelligence. We use language to organize our thoughts, externalize our ideas, and explain our reasoning. But how does our brain decide what to say next? How does the brain drive our conversations forward? The neuroscience of language has largely focused on language comprehension, leaving aside the harder problem of language production. We understand very little about how the brain transforms our thoughts into meaningful, conversational speech. Recent advances in artificial intelligence (AI) and large language models (LLMs) provide a useful testbed: LLMs are the first generation of models that can transform natural-language inputs into meaningful responses and their widespread use in AI chatbots demonstrates an ability to mimic human-like conversational abilities. This project combines generative language models with the creation of a large-scale data set that measures brain activity during human conversation, widely shared with the research community, to model how neural activity guides how we decide what to say in upcoming utterances. This project creates a functional magnetic resonance imaging (fMRI) dataset of interactive conversations: subjects are scanned while they engage in an ongoing dialogue with a partner week-by-week over the course of a year. LLMs are used to model the unique trajectory of each conversation and measure how prior conversational memories impact what subjects ultimately say. Finally, the internal activity of the LLM is mapped onto the human brain activity measured using fMRI to identify signatures of neural activity that drive conversations forward. Specifically, this project quantifies how the brain synthesizes prior words from the conversation into an appropriate response. This naturalistic fMRI dataset serves as a resource and benchmark for the research community. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria. NSF Award ID: 2605721 | Program: 01002627DB NSF RESEARCH & RELATED ACTIVIT | Principal Investigator: Samuel Nastase | Institution: University of Southern California, LOS ANGELES, CA | Award Amount: $2,057,137 View on NSF Award Search: https://www.nsf.gov/awardsearch/show-award/?AWD_ID=2605721 View on Research.gov: https://www.research.gov/awardapi-service/v1/awards/2605721.html