A prototype AI tool that summarizes cancer pathology reports. The tool, developed at Northwestern Medicine, is not yet in clinical use and is undergoing further testing. Image source: Northwestern University
Many open-source AI models generated more complete summaries, especially for molecular findings
AI models can generate more complete summaries of complex cancer pathology reports than physicians, according to a new Northwestern Medicine study that tested six models developed by Meta, Google, DeepSeek and Mistral AI.
The study was published in JCO Clinical Cancer Informatics, a journal from the American Society of Clinical Oncology.
The findings offer a potential fix to a growing challenge in oncology. As biomarker testing expands, and patients live longer, pathology reports have become increasingly detailed and longitudinal, often spanning multiple institutions and requiring clinicians to synthesize large volumes of information under significant time pressure.
In this study, several open-source AI models consistently produced summaries that were more comprehensive than physician-written versions, particularly in capturing molecular and genetic findings that are critical for treatment decisions.
“As cancer care becomes increasingly complex, the burden of synthesizing complex reports is growing rapidly,” said senior study author Dr. Mohamed Abazeed, chair and professor of radiation oncology at Northwestern University Feinberg School of Medicine. “What we’re seeing is that AI can help ensure critical pathological and genomic details are consistently captured — not as a replacement for physicians, but as a tool to augment clinical decision-making.”
Study senior author Dr. Mohamed Abazeed Image source: Northwestern University
The Northwestern investigators analyzed 94 de-identified pathology reports from lung cancer patients. These reports included detailed text describing:
- Histopathological findings (microscopic tumor characteristics)
- Immunohistochemical results (protein expression testing)
- Molecular and genetic data relevant to treatment
The AI models analyzed the text content of these reports and generated structured summaries.
The AI-generated summaries were compared to clinical summaries previously written by physicians. A panel of oncologists assessed each summary for accuracy, completeness, conciseness and potential clinical risk. Across models, AI-generated summaries were consistently rated as more complete, with the largest differences observed in the inclusion of molecular and genomic findings.
“If AI can reliably synthesize these reports, clinicians can review key findings more efficiently, important genetic details are less likely to be overlooked and documentation becomes more standardized,” said study co-author Troy Teo, instructor of radiation oncology at Feinberg. “This could help physicians focus more on patient care.”
The scientists evaluated six open-source language models: Meta’s Llama 3.0, 3.1 and 3.2 models, Google’s Gemma 9B, Mistral 7.2B and DeepSeek-R1. These are not chatbots like ChatGPT, but systems that researchers can download and run locally. According to the study, the strongest performers were DeepSeek and Llama 3.1.
The Northwestern team is now developing an app using Llama 3.1 to eventually allow physicians to upload pathology reports and receive AI-generated summaries for their review. But the study authors emphasize that before deploying the app, they need more testing and validation studies.
The authors said they envision AI as a support layer that enhances, rather than replaces, clinical expertise. It could help highlight key findings, identify missing information and improve consistency in documentation.
Patient reports can span dozens of pages. Even a single missed detail can impact care, and this is where AI may provide meaningful support
Yirong Liu
“Patients with complex cancers might benefit the most,” said study first author Dr. Yirong Liu, a fifth-year resident in radiation oncology at McGaw Medical Center of Northwestern. “In cases where missing a key pathological finding or an actionable genetic marker could change treatment decisions, ensuring that information is consistently captured is critical.”
“Patients are living longer and undergoing repeated biopsies and genetic sequencing,” Liu added. “Their reports can span dozens of pages. Even a single missed detail can impact care, and this is where AI may provide meaningful support.”
The study is titled “Toward Automating the Summarization of Cancer Pathology Reports Using Large Language Models to Improve Clinical Usability.” Troy Teo received funding from the Canadian Institute of Health Research (grant CIHR-472392) and from Amazon Web Services’ Social Impact funding.
No audio available for this article yet.
No quiz available for this article yet.
