7/7 Additional studies which have compared and evaluated UNI (also answering the question, where do better MIL architectures fit in?) 🐍 Ding et al. present GAT-Mamba, which combines graph attention + state space modeling for survival analysis in TCGA-LUAD and NLST-LUAD. Based on UNI features, we see better improvement with more advanced MIL architectures. arXiv: arxiv.org/abs/2406.04377 📊 Following their comparison of ROI foundation models, Chen & Campanella et al. next compared MIL models (same set of tasks as previously). Previously, they found UNI did better on challenging outcome prediction tasks (NSCLC-IO). Can we close the gap further with better MIL architectures? Not by much. Performance gains evaluating TransMIL and Patch-GCN over ABMIL (in ResNet-50 features) do not carry over with UNI. arXiv: arxiv.org/abs/2407.07841 🐈‍⬛ Does this mean we should stop thinking about MIL architectures? Not yet. In PANTHER, we also ablated MIL architectures + pretrained encoder combinations. MIL architectures that benefited the most from UNI were those based on K-means clustering (e.g., AttnMISL and PANTHER). The biggest benefit I see from stronger ROI encoders is better retrieval capabilities, and thus believe that MIL architectures with prototypical formulations (preferably if also pretrainable) will be better for outcome prediction. arXiv: arxiv.org/abs/2405.11643

2/7🐝 Tripathi et al. present Honeybee, a biomedical resource that harmonizes imaging (histopathology, CT), molecular (genomic, transcriptomic, proteomic) and text data (reports) in a vector database. UNI was used as the encoder for embedding histopathology tissue images. arXiv: arxiv.org/abs/2405.07460 ⚡️ All reports, molecular data, and extracted UNI features for all TCGA samples available via HF! huggingface.co/datasets/Lab-… ⚡️ Validated on survival outcome prediction with external validation on lung SCC. arXiv: arxiv.org/abs/2406.08521 💭 Tripathi et al. finds that model efficiency and embedding storage costs (in addition to performance) will become significant factors when implementing retrieval augmented generation (1024-dim vs. 4096-dim features). We agree, and would also like to throw in that retrieval systems may be more sensitive to factors like stain variation, so embedding quality needs to be studied.

5/ 🏥 Breen et al. assess the performance of CPath foundation models on ovarian cancer subtyping, with 12% performance increase in accuracy of UNI over ResNet-50. arXiv: arxiv.org/abs/2405.09990 ⚡️ABMIL heatmaps with UNI vs. ResNet-50 features are also compared. Beyond performance increase, Breen et al. highlights the improved heatmap interpretability with UNI, with greater focus on tumor. 💭 Clinical perspective of pathology foundation models are much in need. Breen et al. appreciated the robustness of UNI in: (1) performing well on 10X patches despite being trained on 20X, and (2) not relying on data augmentation or stain normalization to outperform ResNet-50, but noted the computational costs as a constraint. Foundation models should be able to run on modest computing resources (low barrier to entry for labs that want to go digital), which poses deployment challenges for ViT-Huge/Giant, and potentially also ViT-Large architectures. I agree with this point, and 50/50 use CONCH first for quickly evaluating a task, and follow-up with a larger encoder (e.g., UNI) if better performance is needed.

If you haven't already, please also checkout our #cvpr2023 paper, MI-Zero for performing zero-shot transfer on gigapixel WSIs: arxiv.org/abs/2306.07831