SPIE Conference Part 2: New Study Shows Limited Generalizability of DenseNet Models for Cancer Detection

by Christos Evangelou, MSc, PhD – Medical Writer and Editor

SAN DIEGO, California – In a new study, researchers at Case Western Reserve University and Emory University investigated the cancer detection ability of Dense Convolutional Network (DenseNet) algorithms when applied to different types of prostate tissues. The study showed that the algorithms performed better on their own sample type than on different sample types.

“Our findings indicate that models may not readily generalize between sample types,” said Brennan Flannery, PhD researcher at Case Western Reserve University. “We believe that heterogeneity in tissue morphology limits the performance of DenseNet cancer detection models in prostate biopsies and radical prostatectomies,” he added.

The study findings were presented at SPIE Medical Imaging 2023, which took place in San Diego on February 19–23.

Manual evaluation of hematoxylin and eosin (H&E)-stained tissue biopsies by pathology experts is an integral part of prostate cancer diagnosis and helps inform treatment decisions. Similarly, histological examination of radical prostatectomy tissues can help in making prognostic decisions for individual patients.

Several machine learning (ML) algorithms have been developed to accelerate prostate cancer diagnosis and minimize interobserver variability in H&E image analysis. Because the morphology of prostate biopsies differs from that of radical prostatectomies, ML algorithms are developed using training datasets consisting of either biopsy or radical prostatectomy images. However, whether ML algorithms trained on biopsies generalize to radical prostatectomy tissues remains unknown.

To address this question, researchers used data from the University of Pennsylvania to train cancer detection models. Images were annotated by pathologists for cancer regions and split into tiles of 256 × 256 pixels. Data from 90 patients in Puerto Rico were used for validation.

To assess the generalizability of the models, the team applied the model trained on biopsy data (M^B) on radical prostatectomy images and the model trained on radical prostatectomy data (M^R) on biopsy images. To eliminate batch effects, researchers used data augmentation and stain normalization.

“Because we’re looking at the effect of morphological differences on model performance, we need to take batch effects into consideration to isolate those effects,” Flannery explained.

The study showed that the models performed better on their own sample type (F1 > 0.88) than on different sample types (F1 < 0.65), suggesting that these models do not generalize to each other. Although the combined model trained on both datasets (M^B+R) performed well on both biopsy and radical proctectomy images, it showed low specificity for biopsies.

The researchers believe that reducing the heterogeneity in radical proctectomy training data would be key to improving the performance of the M^R model on biopsies.

“Additional approaches for developing this training set optimization method need to be tested,” Flannery noted. “A training set optimization method needs to be developed for moving a biopsy model to radical prostatectomies,” he added.

The study was supported by various institutions, including the National Cancer Institute and the Cleveland Research Training Network KUH/NIDDK U2C Fellowship Program.