Developing and validating natural language processing tools for memory assessment

Alzheimer's disease is a debilitating condition that affects millions of people worldwide. Understanding the progression of memory decline in Alzheimer's and other memory-related disorders is critical for developing effective treatments and interventions. However, large-scale longitudinal studies of autobiographical memory, which would provide insight into the progression of memory decline, have been difficult to conduct due to the time and effort required to manually score the Autobiographical Interview, the gold-standard method for assessing autobiographical memory. To reduce the scoring burden and enable larger studies on autobiographical memory, PI Dr. van Genugten and OSC Dr. Schacter developed software to automatically score interview responses with natural language processing. Preliminary validation of this AI tool with five datasets shows strong agreement between software-generated and manually generated memory scores. This grant proposes to further develop and rigorously evaluate this AI tool for automated memory scoring. In Aim 1, we will use advanced machine learning methods to improve automated scoring accuracy. Specifically, we will fine-tune state-of-the-art large language models using over 200,000 annotated memory details from existing datasets to enhance performance. To address model limitations, we will use active learning to identify the types of content that the model misclassifies, then augment the training data with additional annotated examples of those types to improve accuracy. In Aim 2, we will develop software for more granular memory scoring. This software will score memories for nine categories of episodic and non-episodic information, including spatial and perceptual details. In Aim 3, we will conduct extensive psychometric analyses to evaluate the validity and reliability of this software, leveraging an existing psychometrics dataset that includes more than 1,300 memories. Finally, we will evaluate automated scoring accuracy on data from across the lifespan and in individuals with dementia, including individuals with Alzheimer's Disease, Semantic Dementia, and Fronto- Temporal Dementia, using eleven existing datasets. Overall, the purpose of this grant is to develop and rigorously evaluate software for automated memory scoring, enabling large-scale studies of autobiographical memory and advancing the ability to study the progression of memory decline in Alzheimer's and other memory-related disorders. Project Number: 1R21AG085061-01A1 | Fiscal Year: 2025 | NIH Institute/Center: National Institute on Aging (NIA) | Principal Investigator: Ruben Van Genugten | Institution: NORTHEASTERN UNIVERSITY, BOSTON, MA | Award Amount: $440,000 | Activity Code: R21 | Study Section: Human Complex Mental Function Study Section[HCMF] View on NIH RePORTER: https://reporter.nih.gov/project-details/11139822