by Christos Evangelou, MSc, PhD – Medical Writer and Editor

In a recent study, researchers from Microsoft Research, Providence Health System, and the University of Washington presented Prov-GigaPath, an artificial intelligence (AI) model that can analyze entire gigapixel pathology slides, paving the way for faster and more accurate diagnosis based on whole-slide images. By leveraging over 1.3 billion pathology image tiles from more than 170,000 whole slides, Prov-GigaPath demonstrates high accuracy in tasks such as cancer subtyping and mutation prediction.+

“Prov-GigaPath is the first whole-slide pathology foundation model that can scale vision transformers to modeling global patterns across the whole-slide image. This opens up exciting opportunities for digital pathology beyond current standard applications such as cancer detection,”

explained Dr. Hoifung Poon, General Manager at Microsoft Health Futures and corresponding author of the study.

The report was published in Nature.

Challenges in Analysis of Large-Scale Real-World Pathology Data

Although digital pathology has shown promise in enhancing cancer diagnostics, the large size of whole-slide images poses significant computational challenges. A standard gigapixel slide may contain tens of thousands of image tiles, making it difficult for existing models to capture both local and global patterns.¹ Previous approaches often resorted to subsampling a small portion of tiles, potentially missing crucial slide-level context.

The researchers identified three key challenges hindering the development and use of pathology foundation models for real-world clinical applications: 1) Scarcity of high-quality, large-scale pathology data for model training; 2) difficulty in designing model architectures that can effectively capture both local and global patterns across whole slides; and 3) limited accessibility of foundation models trained on large-scale real-world patient data.¹

Methodology: Harnessing Big Data and Advanced AI

To address these challenges, the team developed GigaPath, a new vision transformer architecture designed specifically for pretraining on gigapixel pathology slides. GigaPath adapts the recently developed LongNet method to digital pathology, allowing it to scale with tens of thousands of image tiles for slide-level learning.¹

The model was trained on Prov-Path, a large dataset from the Providence health network comprising 1.3 billion image tiles from 171,189 pathology slides. These slides originated from biopsies and resections in more than 30,000 patients, covering 31 major tissue types.¹

The pretraining process involved two stages. First, image-level self-supervised learning was performed using DINOv2 with a standard vision transformer.¹  This was followed by whole-slide-level self-supervised learning using a masked autoencoder with LongNet. The researchers also explored vision-language pretraining by incorporating associated pathology reports for each slide, using contrastive learning to align visual and textual representations.¹

Dr. Poon highlighted the challenges faced during the development process:

“The sheer scale of data presented substantial challenges in setting up the necessary compute infrastructure and conducting pretraining. Digital pathology presents unique challenges to whole-slide modeling, which is central to our innovation in GigaPath, as a single pathology slide comprises billions of pixels and is hundreds of thousands of times larger than standard web images.”

Prov-GigaPath Shows High Performance Across Multiple Tasks

The researchers evaluated Prov-GigaPath on a comprehensive digital pathology benchmark comprising 26 prediction tasks, including nine cancer subtyping tasks and 17 pathomics tasks.¹ The model’s performance was compared against state-of-the-art publicly available pathology foundation models. Prov-GigaPath achieved the best performance on 25 out of 26 tasks, with significant improvement over the second-best method on 18 tasks.

On the TCGA dataset for EGFR mutation prediction, Prov-GigaPath showed a 23.5% improvement in area under the receiver operator characteristic (AUROC) and a 66.4% improvement in area under the precision-recall curve (AUPRC) compared to the second-best model.¹

Dr. Poon noted that the superior capabilities of Prov-GigaPath in discerning morphological features associated with perturbations in key genetic pathways, which is a challenge even the best pathologists grapple with, is a testament to the ability of the model to uncover intricate global patterns at the whole-slide level.

The model outperformed all other models in cancer subtyping for all nine cancer types evaluated, with significant improvements in six cancer types.¹ In addition, Prov-GigaPath exhibited state-of-the-art capability in zero-shot subtyping and mutation prediction tasks when leveraging associated pathology reports.

Dr. Poon emphasized the model’s robustness:

“Our study shows that Prov-GigaPath is an excellent foundation model, attaining state-of-the-art results on a large number of downstream tasks. In particular, Prov-GigaPath outperforms prior state-of-the-art models on many TCGA tasks, which is all the more remarkable given that most of these models were pretrained on TCGA data, whereas Prov-GigaPath has never seen TCGA data during pretraining.”

Looking Ahead

As an open-weight model, Prov-GigaPath allows researchers worldwide to build upon its capabilities, potentially accelerating progress in digital pathology and related fields.¹ By making Prov-GigaPath open-weight, the researchers hope to accelerate progress in the field. Dr. Poon explains, “We hope to enable clinical researchers across the globe to build on top rather than having to start from scratch. We are excited to see tens of thousands of downloads within the first month of release.”

Despite the promising performance of Prov-GigaPath on a wide range of tasks, the authors acknowledge that the scale of the model and dataset requires significant computational resources, which may limit its immediate adoption in some settings.¹ Moreover, additional validation on diverse, real-world datasets is needed.

Dr. Poon noted that a new study from Mount Sinai provided independent external validation, showing that Prov-GigaPath attained top performance on previously unseen Mount Sinai data, especially on difficult gene-mutation prediction applications.²

Dr. Poon is particularly excited about the model’s potential in precision immuno- oncology:

“The most exciting prospect is to apply Prov-GigaPath to even more challenging whole-slide modeling tasks in precision immune oncology, such as tumor microenvironment modeling, immunotherapy prognosis, and complex biomarker discovery.”

References

1. Xu H, Usuyama N, Bagga J, et al. A whole-slide foundation model for digital pathology from real-world data. Nature. 2024;630(8015):181-188. doi:10.1038/s41586-024-07441-w
2. Campanella G, Chen S, Verma R, Zeng J, Stock A, Croken M, Veremis B, Elmas A, Huang KL, Kwan R, Houldsworth J. A clinical benchmark of public self-supervised pathology foundation models. arXiv preprint arXiv:2407.06508. 2024 Jul 9.