Neoadjuvant therapy means months of treatment before surgery. Knowing a patient's chance of pathological complete response (pCR) before treatment could help tailor that treatment. Today we're releasing two preprints on NEO, our AI model that predicts a patient's chance of pCR from a routine biopsy slide. 🧵

The work was done at @Ataraxis_AI by myself, @dhruva_biswas, @joecappadona, Cerise Tang, Ken Zeng, Bartosz Machura, Chuwen Liu, @JanWitowski, and @kjgeras, with our scientific advisor Rich Caruana (Intelligible). Huge thanks to @dhruva_biswas for helping make this work much stronger and for all the writing and refining we did together. We also had support from multiple collaborators: @PTarantinoMD (Dana-Farber), @Omene_CO (Rutgers Cancer Institute), Francisco Esteva (Northwell), Rohit Bhargava (UPMC Magee-Womens), Marcin Braun, Kamila Paździerz, Jakub Czerwiński and Hanna Romańska-Knight (Medical University of Łódź), @AlbertGrinshpun, Bareket Daniel and Michele Buchinger (Shaare Zedek), @fredhow (UChicago), Piotr Wysocki (Jagiellonian University), Brie Chun (Fred Hutch), and Freya Schnabel (NYU Langone). The evaluation in PHERGain and PHERGain-2 was completed with @wearemedsir and our clinical collaborators, including @JavierCortesMD, Antonio Llombart-Cussac, Jose Manuel Pérez-García, Manuel Ruiz Borrego, Peter Schmid, and Marco Colleoni.

Most clinical questions in oncology come with only a few hundred labeled patients, and that will not change soon. NEO suggests a way around it: learn how tissue relates to biology once, from large datasets, then reuse that knowledge for each new question.

Some biopsies capture very little tumor. A biomarker that needs a lot of tissue would leave those patients without a reliable result. NEO barely changed its predictions when given just a sesame-seed-sized area of the slide.

A needle biopsy samples less than one percent of a heterogeneous tumor. A biomarker whose result changes widely with where the needle lands can't be trusted to guide treatment. NEO's predictions stayed consistent across slides, and even individual cores, from the same patient.

For each gene, MORPHEUS was trained only on a single measurement per slide, averaged over a whole tissue sample. It was never told where in the tissue a gene is expressed. Yet, on two public breast cancer slides in which gene activity had been measured at each location, its patch-by-patch predictions tracked the measured ones.

What does MORPHEUS see? Because every NEO prediction rests on the virtual molecular portrait, we asked what biology it captures. Across cancer types, the portraits were most faithful for genes involved in immune activity, stromal remodeling and proliferation: programs that leave visible traces in tissue. For features visible under the microscope, MORPHEUS's estimates matched what pathologists see. Patches it scored lowest for epithelial–mesenchymal transition (EMT) were mostly carcinoma with epithelioid cells; the highest were mostly connective tissue with elongated, spindle-shaped cells.

With MEDSIR, we evaluated NEO in PHERGain and PHERGain-2, two phase II trials that tested whether selected patients with HER2-positive early breast cancer could reduce or skip chemotherapy. In both trials, higher NEO scores meant a higher likelihood of pCR, using thresholds pre-specified for HER2-positive disease.

We evaluated NEO on 1,412 patients from nine independent cohorts that differed in size, subtype mix, response rate and slide preparation: a realistic test of whether the model generalizes. NEO distinguished those who achieved pCR from those who did not far more accurately than four computational tumor-infiltrating lymphocyte (TIL) biomarkers, and it outperformed the Ki-67 proliferation index.

So we split the problem in two. MORPHEUS learns how tissue morphology relates to tumor biology from paired H&E slides and RNA sequencing in 8,742 patients across 32 cancer types. Every patient provides 14,773 gene measurements rather than one outcome label. From the H&E slide alone, MORPHEUS estimates the expression of those genes: a "virtual molecular portrait" of the tumor. NEO then predicts pCR from this portrait and routine clinical variables. Because MORPHEUS has already learned how tissue relates to biology, NEO only needs its pCR labels to relate that biology to response, rather than to learn everything from morphology alone.

A whole-slide image contains millions of cells and thousands of image patches. A response label is a single “yes” or “no” per patient. Learning which of those visual details matter from so few labels, even starting from a powerful pathology foundation model, is objectively difficult.

This clinical task is hard to learn because response prediction is data-poor by nature: each patient needs a pre-treatment biopsy slide and a recorded response at surgery, and fewer than 10% of patients in our database have both. Even so, we assembled 2,492 patients from 14 cohorts across seven countries. Yet by the standards of modern AI, this is still small.