🚀 What do genomic Transformers actually learn about biology? •What knowledge do they hold at random init, after pre‑training, and following fine‑tuning? •We dove deep into every attention head to find out. 📄 Preprint live now “Interpreting Attention Mechanisms in Genomic Transformer Models: A Framework for Biological Insights” 👉 biorxiv.org/content/10.1101/… Code: github.com/meconsens/genome-… ⸻ 🛠 What we built •Scalable mapping between attention heads & biological features (e.g. TSS, GC content, GO terms) •Label‑specific analysis to uncover context‑dependent attention patterns •GPT‑4 summaries for every head’s attention‑feature links •Head ablation experiments to test causal impact on predictions ⸻ 🔍 Key discoveries •Even models with random DNA weights show biologically meaningful heads •Fine-tuning refines, not erases, what pre‑training learned •Tokenization matters: overlapping vs non‑overlapping k‑mers affect interpretability •Heads tied to biology are more predictive than heads with no feature links •Some heads show negative learning—they attend to absence of features ⸻ 🧠 Why this matters We now have tools to ask: what genomic models learn—and which heads are driving predictions. A big step toward truly interpretable, testable genomics AI. ⸻ ⚠️ Limitations to keep in mind •Not every head is interpretable •Attention patterns can be unstable across layers & tokens •Interpretations explain only part—not all—attention variance •GPT‑4 summaries are helpful but can overgeneralize •Results depend heavily on annotation quality & biological context ⸻ TL;DR: We’re bringing interpretability to the core of genomic Transformers—revealing biologically meaningful attention heads, unpacking how tokenization & training shape them, and letting us pinpoint which ones actually matter. 🎉 Huge shoutout to the incredible lead authors in the lab, Mica Consens, Vivian Chu, Ander Diaz-Navarro for driving this forward! @VectorInstitute @UHN_Research @UofT

Exciting News: Our team — Arman (@arman1sa lead, an AI engineer @UHNAIHUB ) + Nasim Abdollahi — placed 1st in the AIRCHECK Hackathon mini-challenge! They built a gradient-boosted model with Bayesian optimization to predict binding of DEL-derived molecules to target proteins. AIRCHECK is a large-scale open-access platform for AI-driven drug discovery, developed by @thesgconline, X-Chem & HitGen, hosting DEL screening data across diverse protein targets. Thanks to @UHN, @Google, and @UHNAIHUB for supporting this work. More to come on accelerating hit discovery with ML! #AI #DrugDiscovery #Cheminformatics #Hackathon

🔥 Unveiling the Future of Genomics with Genome Language Models (gLMs)! 🔥 Our comprehensive review, "Transformers and genome language models," is finally published in Nature Machine Intelligence! ​ Link: nature.com/articles/s42256-0… Key Highlights: 🔬 The Challenges Addressed by gLMs: gLMs tackle the intricate task of interpreting vast genomic sequences, enabling predictions about gene regulation, variant effects, and more.​ 🧠 Transformers in Genomics: Discover how transformer architectures, renowned for their success in natural language processing, are adept at capturing long-range dependencies in genomic data, leading to more accurate models.​ 🚀 Beyond Transformers—Introducing HyenaDNA: Explore innovative architectures like HyenaDNA, which offer efficient long-range genomic sequence modeling at single nucleotide resolution, pushing the boundaries of genomic research.​ 📊 Comparative Analysis of Models: We delve into the evolution from sequence-to-function models like DeepSEA and Enformer to sequence-to-sequence models such as DNABERT and Evo, highlighting their respective strengths and applications.​ ⚡ Strengths, Limitations, & Future Directions: Gain insights into the current capabilities of genomic AI, its limitations, and the promising avenues for future research and application.​ This pivotal work is the result of a collaborative effort led by Micaela E. Consens (@micaelanonsense ), with contributions from Cameron Dufault, Michael Wainberg (@michaelwainberg ), Duncan Forster, Mehran Karimzadeh, Hani Goodarzi (@genophoria ), Fabian J. Theis (@fabian_theis ), Alan Moses. @UHNAIHUB @UHN @VectorInst @uoftoront #Genomics #AI #MachineLearning #Transformers #HyenaDNA #DeepLearning #Bioinformatics #GenomeResearch