The realization of the diagnostic, prognostic, and predictive value of tissue biomarkers has led to an unprecedented rate of development of assays to detect and quantify tissue biomarkers.

This rapid increase in the number of quantitative image analysis methods for tissue biomarkers with potential clinical applications has raised various regulatory issues and necessitates the establishment of appropriate regulatory frameworks. The establishment of such regulations and guidelines will allow the safe implementation of quantitative image analysis assays in patient care.

Regulatory clearance of quantitative image analysis applications

As of 2020, 33 quantitative image analysis applications have been cleared by the US Food and Drug Administration (FDA), allowing them to be sold as commercial kits.¹ Most of these quantitative image analysis applications involve the automated analysis of images for scoring based on biomarker expression in regions of interest selected by a pathologist. Typically, the pathologist needs to confirm the quantification score provided by the algorithm; in case of disagreement, the pathologist can override the score.

According to the Digital Pathology Association and although the FDA has cleared these quantitative image analysis applications, there are no FDA guidelines on the use of quantitative image analysis assays, and many of these algorithms are no longer on the market.¹

Even though most cleared algorithms are based on color detection, more recent applications include algorithms that provide quantification scores based on supervised machine learning (ML) architectures. These ML algorithms can classify tissues into tumoral areas and tumor stroma and quantify biomarker levels in different tumor sites.¹

Regulatory framework and guidelines

Recently, the FDA published an article proposing a regulatory framework issued modifications to AI/ML-based software as a medical device.² The FDA also proposed Good Machine Learning Practices (GMLPs) to guide the use of software in clinical practice.³ GMLPs provide the general principles around data management, algorithm training, and performance.

According to the GMLPs, a Software as a Medical Device (SaMD) Pre-Specification and an Algorithm Change Protocol are needed to describe all modifications necessary for SaMD training and use and how the algorithm can be used safely and effectively.

Good Laboratory Practice (GLP) and Good Clinical Laboratory Practice (GCLP) guidelines are globally adopted standards that regulate all systems used to acquire research or clinical data. These guidelines also ensure the reliability of the data and the ethical integrity of all procedures related to data acquisition. Several aspects of quantitative image analysis applications fall under the specifications set by GLP and GCLP.¹ For example, according to GCLPs, patients must provide informed consent for the use of their data; this requirement also applies to the use of imaging data to train or validate algorithms designed to quantify tissue biomarkers.

Laboratory developed tests and clinical diagnostic tests

Laboratory Developed Tests (LDTs) are in vitro diagnostic (IVD) tests that are developed, manufactured, and used within the same laboratory.⁴ In contrast to FDA-cleared diagnostic tests, LDTs cannot be sold as commercial kits and are not regulated by the FDA.

Because all 33 FDA-cleared quantitative image analysis assays are specific for PR, ER, HER2, Ki-67, or p53, numerous LDTs have been developed to study tissue biomarkers for which there are no FDA-cleared commercially available IVD tests.¹

The Verifying Accurate, Leading‑edge IVCT Development (VALID) Act sets a regulatory pathway for clinical diagnostic tests (including LDTs and IVDs) that are commercially manufactured and sold. Collectively these tests are called in vitro clinical tests (IVCTs).⁵ According to VALID Act, IVCTs that pose a low risk to the patient or user are not subject to premarket approval but still need to follow general controls; high-risk IVCTs remain subject to premarket approval.

Similarly, the European In Vitro Diagnostic Directives into Regulations (IVDR) classify IVDs based on their risk to the patient or user.⁶ IVDR also applies to image analysis algorithms that are intended for clinical use. In contrast to the US, the EU requires post-market surveillance for IVDs and medical devices.

College of American pathologists and clinical laboratory improvement amendments

The College of American Pathologists (CAP) has published guidelines on the quantitative analysis of biomarkers using digital IHC-stained images.⁷ According to these guidelines, the validation of whole slide imaging (WSI) for diagnostic purposes should involve evaluation of concordance with traditional diagnostic methods—preferentially the gold standard method—in an external validation set, documentation of modifications to any components or algorithms of the system, and revalidation upon changes to any element of the system.⁸ Revalidation should also be performed for biomarkers showing heterogeneity in their expression patterns.

According to the Clinical Laboratory Improvement Amendments (CLIA), AI-based technologies should be validated before they are used to analyze clinical samples.⁹ The clinical validation of AI-based technologies aims to ensure that their sensitivity and specificity are equal to or higher than those of manual assessment or surrogate markers. Like with WSI, algorithms should be revalidated in case of alterations to the AI architecture.

To foster standardization of procedures associated with quantitative image analysis of biomarkers in the clinic, the CAP Quality Center assembled an advisory panel of experts to develop clinical guidelines for quantitative analysis of HER2 using IHC images. The advisory panel conducted a systematic literature review and identified 376 relevant articles. Their guideline statements were based on nine studies that had sufficient data and details on quantitative analysis of HER2.^10,11

According to the CAP Quality Center guidelines, quantitative analysis of HER2 using IHC images should be thoroughly validated before clinical implementation to compare their performance to that of validated gold standard methods. Regular maintenance, quality control, and quality assurance are recommended. Moreover, pathologists experts in quantitative image analysis should overlook the use and performance of the technology.

Remaining regulatory issues and future perspectives

Despite the recent publication of GMLPs by the FDA, several issues related to the safe and effective clinical use of ML-based algorithms for quantitative analysis of tissue biomarkers remain to be addressed. For example, the role of healthcare providers during algorithm training remains unclear, and standardized algorithm-training guidelines are lacking.¹

CLIA guidelines recommend revalidation of AI-based technologies if changes are introduced to the AI architecture; however, it remains unclear whether adaptive deep-learning AI algorithms need to be revalidated, as these algorithms are designed to continuously change based on new data.

Another open question related to the safe use of AI-based quantitative image analysis applications in patient care is how pathologists can train complex algorithms to analyze features not readily visible by the human eye.

Apart from HER2, there are no clinical guidelines or statements on quantitative image analysis of other commonly tested biomarkers in pathology practice, including Ki-67, PR, ER, and PD-L1. Such guidelines are needed to standardize procedures associated with quantitative image analysis of biomarkers, including the equipment, reagents, and software used; training provided to users and pathologists; monitoring of system performance; and interpretation and reporting of results.

The establishment of guidelines for quantitative image analysis of tissue biomarkers will ensure the precision and accuracy of the assays when used on patient samples.

References

1. Lara H, Li Z, Abels E, et al. Quantitative Image Analysis for Tissue Biomarker Use: A White Paper From the Digital Pathology Association. Appl Immunohistochem Mol Morphol  AIMM. 2021;29(7):479-493. doi:10.1097/PAI.0000000000000930
2. US Food & Drug Administration (FDA). Proposed Regulatory Framework for Modifications to Artificial Intelligence/Machine Learning-Based Software as a Medical Device.; 2021. https://www.fda.gov/medical-devices/software-medical-device-samd/artificial-intelligence-and-machine-learning-software-medical-device.
3. US Food & Drug Administration (FDA). Good Machine Learning Practice for Medical Device Development: Guiding Principles.; 2021. https://www.fda.gov/medical-devices/software-medical-device-samd/good-machine-learning-practice-medical-device-development-guiding-principles.
4. US Food & Drug Administration (FDA). Laboratory Developed Tests. https://www.fda.gov/medical-devices/in-vitro-diagnostics/laboratory-developed-tests. Published 2018. Accessed November 25, 2021.
5. Diagnostics Reform Heats Back Up with Introduction of the Verifying Accurate Leading-edge IVCT Development Act of 2021. https://www.akingump.com/en/news-insights/diagnostics-reform-heats-back-up-with-introduction-of-the-verifying-accurate-leading-edge-ivct-development-act-of-2021.html. Published 2021. Accessed November 27, 2021.
6. Medical Device Coordination Group. Guidance on Classification Rules for in vitro Diagnostic Medical Devices under Regulation (EU) 2017/746. https://ec.europa.eu/health/sites/default/files/md_sector/docs/md_mdcg_2020_guidance_classification_ivd-md_en.pdf. Published 2020. Accessed November 28, 2021.
7. Aeffner F, Zarella MD, Buchbinder N, et al. Introduction to Digital Image Analysis in Whole-slide Imaging: A White Paper from the Digital Pathology Association. J Pathol Inform. 2019;10:9. doi:10.4103/jpi.jpi_82_18
8. Evans AJ, Brown RW, Bui MM, et al. Validating Whole Slide Imaging Systems for Diagnostic Purposes in Pathology: Guideline Update From the College of American Pathologists in Collaboration With the American Society for Clinical Pathology and the Association for Pathology Informatics. Arch Pathol Lab Med. May 2021. doi:10.5858/arpa.2020-0723-CP
9. US Food & Drug Administration (FDA). Clinical Laboratory Improvement Amendments (CLIA). https://www.fda.gov/medical-devices/ivd-regulatory-assistance/clinical-laboratory-improvement-amendments-clia. Published 2021. Accessed November 27, 2021.
10. Bui MM, Riben MW, Allison KH, et al. Quantitative Image Analysis of Human Epidermal Growth Factor Receptor 2 Immunohistochemistry for Breast Cancer: Guideline From the College of American Pathologists. Arch Pathol Lab Med. 2019;143(10):1180-1195. doi:10.5858/arpa.2018-0378-CP
11. Press MF, Pike MC, Chazin VR, et al. Her-2/neu expression in node-negative breast cancer: direct tissue quantitation by computerized image analysis and association of overexpression with increased risk of recurrent disease. Cancer Res. 1993;53(20):4960-4970.