Providers should invest in and prepare for “clinician-AI interaction,” say researchers who tested OpenAI’s latest LLM preview. Meanwhile, a Mayo Clinic model diagnosed pancreatic cancer before radiologists were able to.

This article was updated on May 7, 2026, with additional information from Mayo Clinic about next steps. 

A pair of recent studies shows big potential for artificial intelligence to detect and diagnose some particularly challenging healthcare conditions.

In one study, Mayo Clinic researchers showed how its Radiomics-based Early Detection Model, or REDMOD, can triple radiologists’ sensitivity in detecting pancreatic cancer at its visually occult pre-diagnostic stage in routine abdominal CT scans. 

The health system said its homegrown AI can find the subtle signs of disease before tumors are visible in routine imaging, “when curative treatment may still be possible.“

Meanwhile, medical researchers at Harvard Medical School, Beth Israel Deaconess Medical Center and Stanford University found that advanced AI significantly outperformed both human physicians and previous large language models (LLMs) in diagnosing complex clinical cases. 

In their new report, they suggest that integrating AI into clinical workflows – such as in emergency rooms – could reduce diagnostic errors when time is tight and information is scarce.

AI found cancer 36 months earlier

Mayo Clinic said its researchers tested their REDMOD AI framework to detect pancreatic ductal adenocarcinoma in real patient imaging and called the results a “milestone” in efforts to achieve earlier detection of one of the deadliest forms of cancer.

“The greatest barrier to saving lives from pancreatic cancer has been our inability to see the disease when it is still curable,”

said Dr. Ajit Goenka, the study’s senior author and a Mayo Clinic radiologist and nuclear medicine specialist, on the health system’s website.

“This AI can now identify the signature of cancer from a normal-appearing pancreas, and it can do so reliably over time and across diverse clinical settings.”

AI-enabled radiomics is the frontier of imaging analytics in oncology. It uses AI and ML to extract high-dimensional data from standard medical images and obtain insights.

Mayo Clinic researchers validated the model on an independent test set, and it identified PDAs at a median lead time of 475 days, they said in their report published in Gut.

They used REDMOD to analyze nearly 2,000 CT scans – including scans from patients later diagnosed with pancreatic cancer that were originally interpreted as normal – and nearly doubled “the detection rate of specialists reviewing the same scans without AI assistance,” Mayo Clinic said.

That’s about 16 months before typical pancreatic cancer diagnoses had been made for the actual patients in the test set. 

REDMOD significantly outperformed human radiologists, demonstrating nearly double the overall sensitivity – 73%, compared to 39% – the researchers said. What’s more, it nearly tripled the sensitivity for cases more than 24 months before clinical diagnosis. 

“The advantage was even greater at earlier time points,” Mayo Clinic said. 

“In scans obtained more than two years before diagnosis, the AI identified nearly three times as many early cancers that would otherwise go undetected.” 

The researchers called the automated, externally validated AI “a necessary step towards shifting the paradigm from late-stage symptomatic diagnosis to proactive pre-clinical interception.” They added: “These attributes position it for prospective validation in high-risk cohorts.”

The path to patient access has already begun, according to Goenka.

“The framework for identifying high-risk individuals already exists in principle,” he told Healthcare IT News on Wednesday. “But translating that framework into practice is harder than it sounds.” 

Mayo Clinic is currently building tools to reliably automate finding the right clinical variables from electronic health records.

“Our eligibility criteria are specific by design, so reaching recruitment milestones requires both precision and scale,” he noted. “That is why expanding beyond a single institution is essential.”

The Mayo Clinic Comprehensive Cancer Center has already opened the first clinical trial in the U.S. to test an AI radiomics algorithm for the early detection of pancreatic cancer, added Dr. Cheryl Willman, Mayo Clinic Cancer Programs’ executive director.

“One hundred patients [ages 50-85] will be accrued in this initial feasibility phase and then the trial will be expanded to Mayo Clinic sites in Arizona and Florida and to other cancer centers and partnering institutions across the United States and the world,” she said by email. 

“Our goal is to reach patients everywhere, within and beyond our walls.”

Advanced LLMs could transform ERs

Researchers at Harvard, Stanford, the Boston-based BIDMC health system and partnering institutions evaluated a preview version of OpenAI’s o1-series model through multiple tests. 

They said they found it highly successful in evaluating challenging clinicopathological cases and clinical vignettes previously tested by a variety of differential diagnosis generators over the last seven decades.

“In particular, the New England Journal of Medicine conference series has been seen as an aspirational goal post, tested by every spanning primitive” since the 1950s, the researchers noted in their report published on April 30 in Science.

They chose diagnostic touchpoints that mirrored “the high-stakes decisions taken in emergency medicine departments, where nurses and clinicians make time-sensitive choices with limited information,” they said.

“Overall, the model outperformed physicians across experiments, including in cases utilizing real and unstructured clinical data taken directly from the health record in an emergency department.” 

The researchers said that more recent studies comparing LLMs like ChatGPT-4 and diagnosis generators like Bayesian systems, symbolic rules-based systems and natural-language symptom checkers also found that LLMs consistently outperformed earlier machine learning systems and the baseline of hundreds of human doctors.

The researchers evaluated the diagnostic and management reasoning capabilities of the advanced OpenAI o1-series LLM across several diagnostic and management reasoning tasks. 

“We further studied LLM second opinions in a blinded fashion against an expert physician baseline on randomly selected patients in a major academic tertiary care emergency department in Boston, Massachusetts,” they said in the report.

Of note, when testing OpenAI’s o1-preview’s performance on NEJM’s clinicopathologic conferences, the model achieved high diagnostic accuracy, finding the correct diagnosis in its differential list 78.3% of the time, the researchers said. 

In 52% of the challenging clinical cases, the model’s first suggestion was the correct diagnosis. But when the LLM included “potentially helpful” or “very close” diagnoses, its accuracy surged to 97.9%, according to the report.

When comparing o1-preview to baseline human physician performance, the model was superior to previous LLM testing benchmarks, outperforming human physician baselines, the researchers said.

In a 70-case comparison with GPT-4, o1-preview outperformed it – 88.6% versus 72.9%. Also, o1-preview performed better than GPT-4 in 24.3% of cases, while GPT-4 only exceeded the new model’s accuracy in 7.1% of cases.

“Our results showed that humans, GPT-4o, and o1 all improved their diagnostic abilities as more information was available; o1 outperformed humans at multiple touchpoints, with the widest gap at initial ER triage, where there is the least information available,” said researchers.

They also said that while their study had limitations, rapid improvements in LLMs hold “substantial implications” for clinical medicine and indicate an urgent need to evaluate the technologies in real-world patient care settings.

“Although applying AI to assist with clinical decision support is sometimes viewed as a high-risk endeavor, greater use of these tools might serve to mitigate the human and financial costs of diagnostic error, delay, and lack of access.” 

Their findings suggest that healthcare systems should prepare to invest in computing infrastructure and design for “clinician-AI interaction that can facilitate the safe integration of AI tools into patient-care workflows,” researchers said. 

“This includes the development of robust monitoring frameworks to oversee the broader implementation of AI clinical decision support systems, monitoring not just final diagnostic accuracy but other metrics crucial for successful deployment, including safety, efficiency and cost.”

This article was first published in HealthcareITNews.

Source: Advanced AI holds promise for high-stakes healthcare, studies show | Healthcare IT News