In the relentless pursuit of breakthroughs in cancer research and therapeutic development, three-dimensional cancer models such as organoids and spheroids have emerged as indispensable tools. These biomimetic constructs faithfully recapitulate the complex heterogeneity and intricate pathophysiology of tumors, all within a controlled in vitro environment. Despite their transformative potential, the technical challenge of non-invasively visualizing these 3D structures over time has persisted. Traditional imaging modalities—predominantly brightfield and fluorescence microscopy—have fallen short in addressing the critical needs for label-free, longitudinal, and high-content imaging. Now, an innovative approach integrating optical coherence photoacoustic microscopy (OC-PAM) with artificial intelligence (AI) promises to overturn these limitations, offering unprecedented insights into tumor dynamics and drug responses at the organoid and single-cell level.

A multidisciplinary team led by Associate Professor Mengyang Liu at the Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, alongside Associate Professor Kristen Meiburger from Politecnico di Torino, have engineered a sophisticated OC-PAM platform specifically tailored for intricate 3D cancer models. By harnessing the complementary imaging capabilities of optical coherence microscopy (OCM) and photoacoustic microscopy (PAM), the researchers achieve label-free, volumetric visualization with remarkable spatial resolution and functional contrast. The integration of AI-driven analytics further empowers the system to perform comprehensive longitudinal tracking, viability assessments, and rare cell detection—capabilities that conventional imaging tools struggle to deliver without perturbation.

A key feat demonstrated by the OC-PAM system is its application to breast cancer organoids subjected to carboplatin chemotherapy. The use of OCM mode enabled high-resolution imaging of individual organoid structures over multiple time points, capturing dynamic volumetric changes associated with drug exposure. Automated algorithms tracked organoid morphology, revealing distinct growth trajectories that delineated responsive subpopulations from a resilient minority exhibiting regrowth—phenotypically consistent with drug-tolerant persister (DTP) cells. This ability to longitudinally monitor tumor heterogeneity and therapeutic resistance in a label-free setting represents a significant methodological leap.

Beyond morphological tracking, the research team introduced a radiomics-based framework that extracts high-dimensional quantitative features from OCM images. Paired with machine learning classifiers, this approach robustly discriminates viable from non-viable organoids without requiring invasive staining or fluorescence markers. The predictive accuracy achieved underscores the transformative potential of combining optical imaging with advanced computational analysis for continuous, non-destructive monitoring of treatment efficacy, a critical unmet need in preclinical oncology research.

The novel capabilities of the combined OC-PAM system extend to probing the cellular microenvironment within densely packed 3D spheroids, particularly the detection of rare cell populations. By co-culturing breast cancer cells with melanoma cells rich in melanin, the PAM component leveraged intrinsic optical absorption contrast to highlight these rare cell proxies amid the stromal milieu. Impressively, the system resolved individual melanoma cells even at extremely low concentrations, offering a sensitive platform for elucidating intratumoral heterogeneity and understanding the roles of minor subclones in cancer progression and drug resistance.

Central to the success of this approach is the synergy between OCM and PAM modalities. OCM provides label-free, volumetric imaging by exploiting the interference of backscattered near-infrared light, facilitating high axial and lateral resolution. Conversely, PAM transduces optically induced ultrasonic waves generated by transient thermoelastic expansion upon light absorption, revealing functional and molecular contrasts invisible to conventional microscopy. Their co-registration within the OC-PAM framework creates a comprehensive multimodal imaging channel that captures both structural and biochemical tumor attributes simultaneously.

The incorporation of AI-based analytics constitutes another pillar of this technological breakthrough. Leveraging convolutional neural networks and advanced radiomic feature extraction, the system automates organoid segmentation, classification, and viability scoring with minimal human intervention. This automated pipeline not only reduces observer bias but also accelerates data throughput, enhancing reproducibility and quantitative rigor for large-scale drug screening endeavors. It exemplifies the profound impact of integrating state-of-the-art optical hardware with computational intelligence in overcoming biological complexity.

Importantly, the ability to track drug-induced changes at the individual organoid level enhances the granularity of therapeutic assessment. Rather than summarizing responses across heterogeneous populations, this system reveals subtle variabilities within clonal populations, capturing early emergence of resistant phenotypes. Such insights are invaluable in informing adaptive therapy regimens and accelerating the identification of novel therapeutic targets specific to resistant cancer niches.

The researchers’ demonstration of non-invasive viability assessment further resonates with clinical aspirations for personalized medicine. By circumventing the need for destructive staining, the platform enables continuous, longitudinal studies on patient-derived organoids, potentially allowing on-demand evaluation of individualized drug responses. This capability aligns with the overarching goal of precision oncology to tailor treatment strategies based on dynamic tumor profiling rather than static, one-time biopsies.

Moreover, the success in detecting rare melanoma cells amidst breast cancer spheroids validates the platform’s application in modeling complex tumor-immune microenvironments and heterogeneous cell interactions. The exquisite sensitivity to minor cell subpopulations could catalyze studies on metastatic colonization, dormancy, and immune evasion—areas critical to unraveling cancer lethality mechanisms.

Taken together, this study establishes optical coherence photoacoustic microscopy combined with AI as a powerful imaging paradigm transcending traditional constraints. With its non-invasive, label-free, high-resolution, and functional imaging capabilities, along with robust computational analyses, OC-PAM is poised to revolutionize fundamental cancer biology research, drug development pipelines, and ultimately, precision oncology clinical workflows.

As translational researchers continue grappling with intratumor heterogeneity and therapeutic resistance, platforms like OC-PAM offer a glimpse into the future of cancer modeling—one where the dynamic interplay of cell populations can be visualized, quantified, and harnessed to devise more effective, personalized interventions. The fusion of cutting-edge optical technologies with AI analytics signals a new dawn in how complex cancer systems are interrogated and understood.

This technological breakthrough exemplifies how interdisciplinary convergence—melding optics, computational science, and cellular biology—can unlock new frontiers in biomedical imaging. The potential for scaling this approach, adapting it across diverse cancer types, and integrating functional assays positions OC-PAM as a cornerstone innovation in the fight against cancer.

In conclusion, the newly developed AI-enhanced optical coherence photoacoustic microscopy platform stands as a versatile, high-impact tool for the imaging and analysis of 3D cancer models. By overcoming the limitations of existing imaging methods, it enables unprecedented, label-free longitudinal studies of tumor organoids and spheroids. Such advancements promise to accelerate therapeutic discovery, illuminate mechanisms of drug resistance, and guide precision oncology with unparalleled resolution and depth.

Article Title: Optical coherence photoacoustic microscopy for 3D cancer model imaging with AI-assisted organoid analysis

Web References: DOI:10.1038/s41377-025-02177-2

Image Credits: Mengyang Liu et al.

Source: AI-Enhanced Optical Coherence Photoacoustic Microscopy Revolutionizes 3D Cancer Model Imaging