The ex vivo microscope could take as little as 30 seconds to look at each specimen and probably will end up being able to diagnose the majority with high accuracy
Interview with Daniel Gareau
Founder and CEO, SurgiVance Inc. and Instructor In Clinical Investigation at The Rockefeller University.
BIOSKETCH: Dr. Gareau is a serial inventor who has designed seven novel medical devices in imaging and spectroscopic diagnostics. He has ongoing collaborations with several academic institutions, and publications in journals including Cell, which have received over 1,500 citations. Dr. Gareau pioneered the field of ex vivo confocal microscopy for pathology by developing an imager (Gareau, Li et al. 2008) that first achieved high pathology accuracy (Karen, Gareau et al. 2009). His translational breakthrough to mimic the appearance of standard pathology (Gareau 2009) is recognized (Krishnamurthy, Brown et al. 2019, Malvehy, Perez-Anker et al. 2020) as having nucleated the “next-generation digital microscopy” and “a revolution in fast pathology.”
Interview by Jonathon Tunstall – 15 Oct 2021
Published – 20 Jan 2022
JT – Dr. Gareau, welcome to Pathology News. Can you tell me something about your background and how you came to be involved in medical imaging?
DG – At the turn of the century, I entered college as a music major and quite soon, started asking questions about what is going on with this complex cellular system buried deep in our heads. I understood that Minsky invented the confocal microscope to see into living tissues such as brain tissue, and that led me to ultimately to confocal microscopy. So, I did my PhD in ‘in vivo confocal microscopy in turbid media,’ and the skin was my most accessible target. From there, the major application seemed to be cancer detection so I got into that and here we are.
JT – So you jumped directly from music to confocal microscopy.
DG – Well music immediately led me to the question ‘what is going on’ and that led me to Minsky’s invention which provided cellular resolution in living brain slabs. So, it was premature to study music because I needed to first go back to Minsky’s invention to understand what was going on inside the brain. Then as a graduate student, the biological sample that you have easiest access to is the skin, so I started doing confocal on skin. Then a gentleman by the name of Glenn Merlino introduced me to his mice that had melanoma and so at OHSU, I could say that my clinical achievement was to be the first to non-invasively detect pagetoid melanocytes using endogenous reflectance confocal microscopy. That is an endogenous contrast based on the microscopic and nanoscopic refractive index variations and I devised a numerical simulation that rolled a dice many times and translated the random numbers into photon trajectories. It was a Monte Carlo model that I adapted to focus light into living tissue.
JT – So that means a random pulse of light based on the dice model?
DG – Yes, it’s a standard technique called Monte Carlo, but my contribution was that I modelled confocal microscopy with it, for perhaps the first time. I showed that the confocal endogenous reflectance signal could be used to extract the optical properties of the tissue; specifically, the scattering properties, including the scattering coefficient measured in inverse centimetres and the scattering anisotropy, which is a unitless measure of the directionality of the scattered light.
JT – So why was it important to use a random model?
DG – The random model represents random photons in the laser. For instance, if you know your laser beam has a Gaussian profile and you take every surface area element under that Gaussian profile and you assign each element a number from one to one hundred, then you draw numbers from one to one hundred and launch them at the positions indicated. Then if you do that billions of times, you get a smooth light pattern of the focussing light into the biological tissue. So, the random numbers were really just a way to draw on probability density functions such as a Gaussian probability of irradiation.
JT – So this could for example be used to mimic natural exposure to sunlight?
DG – Yes, my Ph.D supervisor, Steve Jacques, is famous for that, but it was me that realised that we could use the same photon transport simulation to transform confocal measurements to tissue optical property measurements which are otherwise impossible to make without a transmission system. That was the technical portion of my thesis, but let’s talk about the clinical impact, because I was doing in vivo imaging initially and then I moved to Sloan Kettering because I wanted to make confocal microscopes from scratch.
JT – So far we have been discussing your work at Oregon Health?
DG – Yes, I graduated doing that work on detecting pagetoid cells in mice. That shows that at least it can work for a mouse model of melanoma. I did the number crunching stuff at Oregon Health, but it was the hands-on construction of novel confocal imaging systems that got me to my post doc at Sloane Kettering. That’s where things get exciting because that’s where the stars in the sky story begins. I was looking at the same old endogenous contrast we had always used, and the same endogenous contrast that was being used when I got to my post doc job, and I thought to myself, ‘not specific enough.’ So, I started asking the question, ‘how is it done in histology?’ Haematoxylin is a stain that is very specific to nuclei, so knowing how to tweak microscopes, I switched the filters, found a stain, stained the nuclei, and then all the cells popped off the screen like stars in the night sky with super high contrast. Then I wondered, ‘what does this get us?’ and the answer of course was that you could see these infiltrative strands of BCC. Even these days, for things that are really difficult, like perineural invasion which has a prognostic implication, fluorescence specificity is really important. So, that’s the first thing I did, just asking ‘how can we make this sensitive and specific for basal cell carcinoma detection when the tumors are these challenging types.’
JT – I guess you would have the ability with this type of technology to look at the tumor boundary, the invasive part.
DG – Well to be able to do pathology essentially. Anything you can do by looking at nuclei with haematoxylin you can do by viewing nuclei with acridine orange staining, and that works in the confocal in fluorescence mode. So, in fact, I was adding a mode, and then at least in the hands of our colleagues, the Mohs surgeons, they can see the nuclei and then make the mental leap that they could do pathology by looking at grey scale images. All of a sudden, the acridine orange gave a contrast so that they could really detect it accurately. That was the great moment, but it was short lived, because then I tried to take this and show it to other people and I had to explain every time ‘this is a grey scale image, the black and white pixels are the fluorescence label that show you the nuclei.’ All the pathologists that were patient enough to wade through that explanation really bought into it, but that was only about 5% of the clinical and translational audience. Everybody else was skeptical and just saw it as a research tool.
So, in fact, I had turned away what everyone was working on, reflectance mode, to get to this fluorescence mode, and had I encountered the problem where people are not understanding what they are looking at. I knew that I needed to create something that was visually intuitive and then I remembered the reflectance mode and I realised that a change was needed. You see everyone back then was using aceto-whitening, which is the application of acetic acid or vinegar to make the nuclei look very bright. What that actually does, it condenses the chromatin filaments into aggregates so that the scattered light increases, and you can better see the nuclei. The result is very bright nuclei on a bright background. For counter-contrast to the fluorescence nuclear staining, I didn’t really care about the bright nuclei, so I omitted the aceto-whitening step and used reflectance to mimic eosin, taking the hint that eosin is a counter contrast for haematoxylin. In fact, Eosin stains everything and the purple is so strong that it is not technically accurate to say that eosin is a counter contrast, but for practical purposes it is. That led me to a quantum leap for this technology which was to color code the images to look like haematoxylin and Eosin.
I immediately knew that we had something because it didn’t need explanation anymore. The first person that I showed it to, I asked ‘what do you see here?’ He immediately said, ‘here is an endocrine gland, this is epidermis, there is a basal call carcinoma tumor.’ Then I asked, ‘how is this image made?’ And he said ‘well it’s H&E’. Over the course of the past 15 years, this colorizing has become mainstream.
JT – So, what you have there is a 3-D high resolution version of H&E.
DG – Well I would prefer to call it a “DIRECT-TO-DIGITAL” confocal version of H&E, because the resolution depends on what sort of objective lens you use. I think of it as a confocal image equivalent to H&E, and it is specifically equivalent in the visual appearance of the image. A decade later it had become more mainstream, then a decade later still, AI caught up to it and went the rest of the way. Now you can colorize multi-modal confocal images to look exactly like H&E and then that brings up all the ethical questions of whether the AI is adding content to the images and whether that is ethical. That sort of closes that story because the original discovery was to transform multi-modal confocal images including endogenous unstained reflectance mode to mimic the appearance of haematoxylin and eosin. By the way, ‘to mimic’, is the operative word. and as a disclosure, that is a verbatim claim on my intellectual property.
JT – The reason for mimicking H&E then is purely for pathologist satisfaction? The use of those particular colors (because they could be any colors) is important because they are the colors that a pathologist is used to seeing.
DG – Yes exactly. You could say that it co-opts the underlying visual language of pathology. It’s a conditioned stimulus because the pathology field is conditioned to see and to understand and to think this way. So, it is literally a translation of images, and once you do that, you enable human vision. You may be thinking ‘what’s the relevance in the AI world?’ Well, the relevance is that if you want to paint an automated AI diagnostic, you need to paint it on a canvas. This is the canvas, the H&E colors, the pink and the blue, because for this generation of pathologists, that’s what they are used to. You can even show them that you can put it into blue and yellow, which gives greater contrast and may allow them to discern more detail, but how many other colors do you think they want to see?
JT – Well, they are used to seeing other color palettes through IHC staining.
DG – But what part of their daily lives is IHC staining? It’s like a couple of percent and I’m not saying that the future is going to be the way it is today, because in general, just because things have been a certain way, it doesn’t mean that’s the way it needs to be or should be. This is value add and the changing of things is tough logic.
JT – So that’s a good explanation of the technology. Let’s talk through the real life benefits then, the pros and cons. For example, the cost per test, because H&E is as cheap as chips.
DG – I think that it doesn’t necessarily cost more than H&E. Sure, H&E is pretty cheap, but I would say that we are not trying to change the cost of anything right now. We do add a whole bunch of values somewhere down the line, but we are in the disruptive business, only as and when we need to be. You have to work from within a system when people’s lives depend on it. So, if could tell you that each additional confocal analysis with full AI was going to cost a femto penny, like each google search (which by the way, is a great model for diagnostics because we want to do things really inexpensively and do them with big data), but we are not in the business of rocking boats. We are in the business of being disruptive at a key point. So, with the pre-statement of ‘we are not going to change anything’ we will say that the advantages of confocal pathology or ex vivo pathology have not yet even been fully defined. I would say this novel technology has the potential to broaden the bandwidth of pathology, more pixels, more images, more detail. All this may become very important training data for AI in the future.
How should we define ex vivo pathology?
DG – Ex vivo pathology is when you image a fresh tissue specimen which is not frozen or fixed with a rapid preparation and a digital imaging device. The rapid preparation is staining, and the digital imaging device is a confocal microscope or a MUSE device. MUSE is surface UV illumination. What we are going to call an ex-vivo microscope is any one of a new wave of small, advanced imaging systems that do 3D imaging and get you an image digitally. So, ex vivo pathology is the process of imaging fresh tissue on a novel digital imaging platform and I believe we have developed the preferred platform right here.
The potential benefits of this are that if you think about not changing the cost model, it just increases the band width of pathology. That’s the quote to get someone to understand this technology. The band width can be defined as the number of pathological evaluations times the number of cells per pathological evaluation. Once you have that, you can analyse more band width on one case or you can analyse more cases with the confocal method. If you just considered a typical frozen section where we evaluate four slices on a slide, with a confocal you can get 20 or 100 unique sections and have them available instantly in a digital format. That is maybe 100GB of information and it is superior in band width to what the current vendors have done (much to their credit) in digitizing pathology.
JT – We could make a general statement that current digital pathology represents a single plane and a slice of tissue that is 3 or 4 microns thick.
DG – What we do to enable ourselves to speak the same language, is that we take the 3D confocal output and we apply some intelligent math to reduce that to a 2D that has full tissue coverage at maximum resolution. it’s about translating the novel stuff into the current language and then taking it forward. The advantages are that one, it can be faster and two, it can be higher band width, and these have both qualitative and quantitative approvals. Quantitative can be in most surgery situations where the number of minutes that you delay is important and that also includes some cancer resections. Quantity can also be the difference between evaluating on one slide and evaluating on several slides to form a 3D date set. Now there is a question, a fair question as to how much value is added by quantity in the second sense of analyzing a 3D image. You could say a 2D image is all that we need and that our company and several other companies are just trying to find a sales point for the value add of 3D pathology. But any way you break it down, the ex vivo microscope increases the band width of pathology. It’s instant (we can do a Mohs section in a couple of minutes) and then it can be read by anyone in the world. That’s important, and those are all the quantitative improvements but there are several qualitative improvements too. For example, imaging on fresh tissue doesn’t suffer freezing artifacts and it doesn’t degrade RNA for multi-omic analysis. It also doesn’t prevent you from culturing the cells from that specimen in a biological twin of the disease that’s in a lab. In short, ex vivo microscopy is really the platform that will set all biosciences free when it comes to treatment of disease, because a treatment of disease requires a diagnosis of a disease, and a diagnosis of disease has until today required standard histology. So, when you change to instant digital pathology, you not only get the quantitative improvements, but you get all these qualitative improvements that will unfold over the coming decades. Another one of the qualitative improvements is that there is a lot of information in the unstained reflectance confocal signal that has not yet been evaluated. That is because it’s hard to do with the human brain. I predict that with the advent of AI, this information content will start adding the value in pathology that we have started to see with the eye. For example, there are publications that show that small strands of tumors are visible in reflectance confocal but not in the histopathology. That is a qualitative difference where you can get something with the ex vivo pathology which you cannot get with the histology.
JT – I like the term ‘Instant Digital Pathology’ that is very powerful.
DG – Yes because no one has time to wait around. Time is money. “Direct-to-digital” is another good term.
JT – We also have a shortage of tissue pathologists at the moment and workload is increasing.
DG – We need people, but we also need the machines to make people’s work more interesting and efficient. So that brings us to the elephant in the room, which is the role of AI in pathology? Now there is no way, that we are going to let the value add of that enormous elephant go. That is just the way it’s moving. So, the key is to use the AI to provide diagnostic nudges that remove all the uninteresting work that the pathologists do. I’m not suggesting that there are any pathologists that are uninterested in their work, pathology is fascinating. However, if you can dynamically reallocate your work to the most fascinating part, why wouldn’t you? For example, nobody needs to be reading nodular BCCs, so if a machine can do it and the relevant regulatory bodies approve that — which they haven’t yet but hopefully will – than why not do it.
JT – Or counting cells.
DG – Right, and the data reduction powers of AI will certainly be transformative.
JT – Agreed. AI is a digital assistant for the pathologist rather than a replacement.
DG – Now that doesn’t just hold in cellular pathology, it also holds in clinical pathology. Clinical dermoscopy, and the adoption of AI as an adjunct, is not only appealing to the humans who are in charge of caring for a person, but it is also an attractive regulatory standpoint in a company that is concerned about the regulatory impact of the commercialization of AI. Those are two really important paths forward for AI.
JT – Let’s talk about how this technology integrates into the workflow. Let’s assume that there is a future world of ex vivo in conjunction with AI, where does that exist? That must be in the large medical centers.
DG – The way I see it is that there will be two phases. There will be a transition phase for confocal and AI. Let’s group them together from now on, confocal is the eyeball and AI is the brain which acts as your digital pathology assistant. Let’s also assume that there is going to be some product from a company such as SurgiVance or someone else. When a biopsy is taken it will always go through confocal and the diagnostic AI will always be run and at least in the transition phase, that specimen will always go on to standard histopathology. In the longer term, the permanent phase will be in place when some or all of those diagnoses are being made by the digital assistant and the pathologist is signing off on the suggestions from the AI.
JT – So you are suggesting that we will no longer need the backup of the FFPE tissue sample?
DG – Right, but that’s not today, that’s the future. Here’s the thing though, the future is coming soon, because we have already proven the principle. By that I mean both in the research world and in the regulatory world, so the transition phase will be concluded with the conclusion of this proof of principle. Eventually, we will all spend our time on the 20% of cases that don’t have high certainty with the AI. The AI can start acting as an adjunct as long as it has a good certainty readout, and then eventually it will replace the common diagnoses for sign off and lead to more complex diagnoses than a human could ever make without using this new diagnostic aid.
JT – You are presuming that we will discover digital biomarkers that you can only detect through AI.
DG – Yes, so imaging biomarkers is one term and pathology bandwidth is another. I already defined pathology bandwidth and an important add on is that the number of cells you can evaluate becomes a lot greater with AI.
JT – They can only be evaluated by AI itself
DG – Right, but it is not just limited to AI because an imaging biomarker could, for example, be the display AI puts out to the human which then becomes the diagnostic nudge. That is what we want it to do as an imaging biomarker.
JT – So, the information could be given to the human within boundaries that the human can observe and detect but it could initially be undetectable to the human eye.
DG – That is true and that is sort of how the machine can teach the human to understand imaging biomarkers, though I’m not really satisfied with my definition of imaging biomarkers.
JT – I would define it as a nuance within the image which is too subtle for a human to detect, but with computer vision you can detect that element and present it. You can also find it across multiple images, and it has significance.
DG – And you can display it.
JT – Yes, you would have to display it or at least show it as a number or some sort of image that a human could understand, but the key is that it has diagnostic significance.
DG – In fact, our group is a little more advanced when it come to the imaging biomarkers for dermoscopy as that is an easy language for us to translate. We have only one confocal imaging biomarker for tumor positivity now. Things are moving quickly.
JT – So imagine a young pathologist in 15 years’ time, maybe less than that, and we already see this happening. There are going to be labs which are still using microscopes, labs which haven’t digitized for one reason or another, whether it is cost, or they are too small or maybe they are staffed by older more skeptical pathologists. What would be more attractive to you as a young pathologist joining this field? I think you are going to want to work with the latest technology and maybe that includes ex vivo. So, what happens ultimately to the pathology labs that are still based on standard microscopy?
DG – Well, they can use ex vivo microscopes as an adjunct to standard microscopy. Say you’re a pathologist and you do 800 slides a day. The ex vivo microscope takes 30 seconds to look at each specimen and it can diagnose 80% of your specimens with AI with 99.9% accuracy. You, the pathologist can then take the rest if the day off, or more importantly you can spend your time on other things. You still need histopathology because you are going to need special stains, immunohistochemistry etc., but your bread-and-butter stuff can be expedited. You are not doing less work, you are just working smarter, and we are taking great pains to make everything as understandable as possible. So, in 15 years, the new generation of pathologist will be reading digitally stained slides. I still think they are going to be reading slides in a way that is visually the same as today’s reading, but they are also going to be wanting all those advanced imaging biomarkers that point out things that the machine can see, that the human can’t see easily. However, if you’re a pathology lab that has a million dollars in infrastructure, you’re not going to have to buy that latest gear from your standard supplier and instead of spending let’s say $80,000 to get the latest microscope, you’re going to keep your standard microscope and you’ll still be able to produce your digital slides. your scanning instrument will still be there, but you are going to have an adjunct ex vivo imager that’s going to de-bulk your work with those other systems and make your overall operation a lot easier. The pathology lab the pathologist and the treating physician taking the biopsy are all equal partners. The Mohs surgeon will use ex vivo to replace frozen sections whereas the laboratory will use this to replace permanent sections on the bulk of the specimens that they no longer have to dedicate opportunity cost to. It’s great for pathology labs because they can do a 30 second procedure on a specimen before the standard procedure and that may save them 25 minutes on their standard procedure if the AI can hit the nail on the head. Of course to manage patient care, regulatory factors are absolutely critical and this type of technology is not yet approved. Hopefully it will be soon.
JT – Let’s talk then about Surgivance.
DG – Surgivance stands for advanced surgery, and what we are developing is advanced surgical pathology which has a hardware component and a software component. The hardware component is the best possible ex vivo microscope. There are companies out there that are making ex vivo microscopes, but you want one that has no moving optics, and you want one that is the least costly given that it is capable of cellular resolution.
JT – And should it be small?
DG – Yes, the prototype now is about 12 inches wide by around 18 inches tall and this could be made the size of a mobile phone.
JT – That’s because you have to get this near to the surgery.
DG – It’s at the point of biopsy in a dermatologist’s office. And it’s not imaging on the patient because we want to reserve the right to do dangerous things to the tissue like stain it with toxic dyes that enable high contrast imaging. It’s all about finding the biggest value add for the least cost. That’s where surgery ends and where the hardware is located now. The value add, is that it is capable of cellular resolution because that’s what you need for pathology. The way the Surgivance hardware works is that it is sort of a cross between a confocal microscope and a Xerox machine so it focusses a line of light and moves the sample across it like a Xerox machine would do and because it has no moving laser beams like a standard confocal microscope, you can manufacture it in a rigid metal case. That is our plan. It is essentially like one huge objective lens that’s in a metal case. and this is the minimum technical complexity to get cellular resolution. That is the first part of our company, and it has been SBIR funded to develop this imager / ex vivo microscope. So, one half of our company we already got developed and that is a hardware compacted ex vivo microscope, which is the preferred ex vivo microscope for scaling the band width of pathology.
JT – So you have a patent for that?
DG – Yes, anyone can Google LSSCM (Line Scanning Stage Scanning Confocal Microscope) and they will get the paper that describes the technology that was patented.
The second part of our company is the software patent that covers the colorization of multi-modal confocal microscopy wherein one mode is unstained reflectance mode to mimic the appearance of haematoxylin and eosin. That’s the claim, and anybody who colorizes multi-modal confocal microscopy including the unstained endogenous reflectance mode to mimic the appearance of haematoxylin and eosin would be violating the claim in our patent. We’ve raised about $1.5M to date and we just got awarded our second SBIR award from the National Science Foundation of the USA so further investment is coming shortly.
JT – Are you at the prototype stage? Do you have and instrument installed in a clinic somewhere for evaluation?
DG – No, right now we are in the stage where we have a protype here in the start-up company and we have an agreement with Northwell Health where they send specimens to our prototype for imaging Those studies are ongoing right now. We hope to get into prospective studies in the coming year or two.
JT – So we could describe those studies as proof-of-concept studies?
DG – Yes, I would say that is a fair assessment, but Surgivance has another device which is a software device, and that device, which is just now getting funding, will in the very short term have the capability to provide diagnostic nudges for pathology images including confocal digitally stained images. That product, which is aimed for launch within 6 months will be a software that lets you convert confocal images into an H&E equivalent and then run the diagnostic nudge evaluator which will present the imaging biomarkers to do the pathology for you. Before we have regulatory approval, it will come with a disclaimer that this technology is not to be used to manage patient care. Eventually we want this preferred ex vivo imager to be the size of a mobile phone and to have both the ability to have H&E appearance and to provide automated pathological evaluation as well as instant digital connection to the electronic medical record. The uses for this would be several fold. One of them would be in the case of Mohs surgery where digital connection to a remote pathologist is not as important because the pathologist is the Mohs surgeon. A second application of this could be in a dermatologist’s office, where again the workflow would be totally changed, because it is no longer necessary to wait a week for a diagnosis. Diagnosis can be made in the same visit. So in Mohs surgery it is about having a margin screening assessment in 2 minutes by the patient bedside.
That part will be a 510K approval and our intent is to file for a 510K application for the indication of margin screening, whereas our intent would be a de novo application in dermatology for the assistance of primary diagnosis from a biopsy, a shave biopsy or a core biopsy.
Then we have other markets in digital pathology which range everywhere from urban centers to streamline lab bulk imaging procedures to specialised intraoperative cases such as oesophageal resection where margin status is important. We have surgeons in pancreatic cancer resection that are convinced that evaluating margins more accurately for pancreatic resections will improve the survival rate of pancreatic patients. Pancreatic cancer is the exact opposite of skin cancer, being not very prevalent but very deadly. So, this is our roadmap in a sort of telescoping order of magnitude.
JT – there must be a market for ex vivo in pretty much any cancer diagnosis?
DG – Yes, that is true, and I also didn’t mention the research market. It all comes down to pathology and if we are going to identify pathology like cellular pathology, the way we have been doing it for a hundred years as the key to digital health. Then we also have to recognise it as the key to biological understanding as well. That is because it is necessary to analyze complex cellular populations and so that impacts neuroscience, all these fields.
JT – Dr, Gareau, we’ll leave it there. Thank you for your time today.
