Vision-Language Models with Explainability for Breast Cancer Prognosis using Image and Clinical Data.

Breast cancer is the most prevalent cancer among U.S. women, accounting for nearly one-third of new cancer diagnoses in women annually.2. Accurately predicting recurrence and mortality is critical for prognosis, treatment decisions, and improving patient outcomes. However, current clinical risk models, such as the Nottingham Index and PREDICT Breast,4,5,23,107 have limitations that hinder precision and clinical utility. More comprehensive prognostic tools are needed. Digital pathology imaging offers a promising solution by capturing tumor characteristics beyond human visual discrimination, including nuclear heterogeneity and chromatin complexity. Recent analytic advancements suggest that pathology images may capture nuanced tumor structural data beyond the discriminating capabilities of traditional microscopic evaluation by humans, offering novel parameters for refining risk prediction. This proposal aims to develop a robust Survivor Model (Aim 2) that merges whole slide image (WSI) data with traditional clinical prognostic factors to improve survival estimation. This will be complemented by developing an Explainer Model (Aim 3) to enhance transparency by generating pathologist-interpretable text descriptions of predictive image features. Our specific aims are: AIM 1. Establish benchmarks for predicting recurrence and survival at 5, 10, and 15 years using a clinical epidemiological model and a deep learning image-based model. 1a) Assess the PREDICT prognostic model107 performance using clinical data. 1b) Retrain and evaluate the SlideGraph neural network model24 using image data. AIM 2. Develop a new Survivor Model by expanding QuiltNet,25 our team’s vision-language model, to integrate clinical features and WSI data for recurrence and survival prediction, then compare it against benchmarks from Aim 1. AIM 3. Develop an Explainer Model for WSI-based survival models incorporating pathologist feedback. 3a) Create an Explainer Model that interprets survival predictions from both the Survivor Model and another survival model. 3b) Integrate pathologist feedback to refine predictions. 3c) Validate the Explainer on CALGB and the large, diverse PATHWAYS Study dataset. This work will generate a clinically actionable risk prediction tool that combines deep learning with pathologist- informed reasoning, advancing precision oncology and enhancing breast cancer prognosis. By improving risk stratification and interpretability, this model has the potential to guide treatment decisions, personalize patient care, and ultimately improve outcomes for individuals with breast cancer. Project Number: 1R01CA299626-01A1 | Fiscal Year: 2026 | NIH Institute/Center: National Cancer Institute (NCI) | Principal Investigator: JOANN ELMORE (+2 co-PIs) | Institution: UNIVERSITY OF CALIFORNIA LOS ANGELES, LOS ANGELES, CA | Award Amount: $825,627 | Activity Code: R01 | Study Section: Clinical Data Management and Analysis Study Section[CDMA] View on NIH RePORTER: https://reporter.nih.gov/project-details/11296299