Pinned Tweet
If you’ve ever
- thought AI protein folding is magical ✨
- wanted more than a pLDDT score 🔎
- or just think mech interp in bio is cool 🤓
then read the 🧵 👇 on our first paper towards interpretable protein structure prediction just accepted to workshops at ICLR
A First Step Towards Interpretable Protein Structure Prediction
With SAEFold, we enable mechanistic interpretability on ESMFold, a protein structure prediction model, for the first time.
Watch @NithinParsan demo a case study here w/ links for paper & open-source code 👇
text compression - bits/sentence
dna compression - bits/genome
same math for both
There has been some rather impressive progress on the Hutter prize for text compression recently (~10%; a huge trend break). Interestingly, it's all based on the idea of training / overfitting a small transformer on the data to be compressed. This model will assign high likelihood to the data, and consequently arithmetic coding (using said model) will assign short code lengths to the tokens. Because the model itself is not known by the receiver, its weights must be compressed and included in the code length as well, and so you train with a loss that includes that cost as well.
All the way back in 2020/2021 when I was working on generative models for compression, we explored this idea for video compression: take a pre-trained video compression model (discrete-latent AE), and overfit it on one video using a loss L = coding cost of weight-delta + coding cost of data given model (i.e. log likelihood). We used a spike-and slab prior on the weight delta so that it is cheap to encode "no change" to the pre-trained weights.
We found it to be very effective, but needing to train a model at encoding time + inferencing it at decoding time is kind of a hassle, so we didn't pursue it further. It still feels like a long shot, but given how competitive such approaches now appear to be, and how robust (quantization-aware) training techniques have gotten, it may be worth another look. Especially for cases where encoding compute is not a major concern but code length is (i.e. content on any platform with lots of users, where you encode once and transmit + decode many times).
Incidentally, compression is also an excellent task for research on coding / AI research agents. You have essentially infinite training data, an un-hackable reward, and according to Kolmogorov / Chaitin it will never saturate.
References
@tivaro @IamHuijben Cohen, Overfitting for Fun and Profit: Instance-Adaptive Data Compression arxiv.org/pdf/2101.08687
@OriYoran The KoLMogorov Test: Compression by Code Generation arxiv.org/abs/2503.13992
Hutter Prize (Marcus Hutter, @mhutter42): prize.hutter1.net/
Matt Mahoney, Large Text Compression Benchmark: mattmahoney.net/dc/text.html
Hutter prize SOTA over time
epoch.ai/data/hutter-prize?v…
Recent entries
fx2-cmix-transformer, Vladimir Ivanov (the award winner; offline-trained transformer, 4-bit weights shipped): github.com/astOwOlfo/fx2-cmi…
zmix, James Byrne (adds the coded size of the weights to the training loss): zmix.frostbyrne.io/v1.0/writ…
cmix-neif-beehive, Vladimir Malinovskii (differentiable rate term on the weights, with ablation): github.com/galqiwi/cmix-neif…
cmix-neif 2.0, Marcus Wölk (best entry so far; entropy-coded weights with bits-back): github.com/marwoelk/cmix-nei…
tufazip @tufalabs, Tommy He / Tufa Labs (shipped prior plus online adaptation; analysis of model bits vs. data bits): tufalabs.ai/research/tufazip…
DNA-4, Ao, Sun & Sun (RWKV-7; arithmetic-coded 4-bit weights, then online fine-tuning): github.com/ziq-ao/DNA4-RWKV
49
sequencing a human genome now costs ~$100 in raw reagents at scale.
at these prices, the billion-dollar prenatal genetic testing industry is about to change dramatically.
and whole-genome sequencing will unlock breakthroughs in what AI can predict about disease. 🧵
1
4
182
why now? why hasn't this been done yet?
- genetic data hard to access. i could only get bc of nature paper i co-authored in HS
- 1M+ whole genomes got released in the last 2 years
- good GPUs only became available on the biobanks last year
1
12
mit ai for bio 🚀🚀🚀
We’ve raised $110M to bring AI-powered biomolecular design to every scientist!
Boltz’s Series A is led by @BessemerVP, following 50-fold growth in molecular token usage from our users since June and partnerships with 6 of the world’s 20 largest pharma companies.
alphagenome 2 is great work. but surprisingly it only ever sees one genome
predicting who gets heart disease or prostate cancer requires training on millions of genomes
hoping alphagenome 3 trains on more than one
AlphaGenome 2 is live! A step change in performance and new capabilities over AG1! @adamgayoso's thread covers the key results.
Manuscript: storage.googleapis.com/deepm…
Blog: deepmind.google/blog/alphage…
API: github.com/google-deepmind/a…
A few notes under the hood & what's next:
scaling laws for human genome pretraining soon
Genomes are long but redundant. Not great for pretraining.
To fix that, we've developed a lossless tokenization strategy that shortens sentence length by >2000x.
For pretraining, we achieve a 0.55 validation CE on All of Us.
Public replication on 1KG linked below.
17
744
lossless compression for human genomes?
richard hendricks would be proud
the failure of Lp(a) drugs shows why we need interpretability in drug discovery.
heart disease is heritable, but thousands of genes interact to cause heart attack.
genomic language models learn these interactions. interpreting them will reveal new drug targets.
Massive failure at Novartis to improve cardiovascular outcomes by lowering Lp(a). What went wrong? What now? Is Lp(a) useful? lifespan.com/blogs/magazine/…
106
fda finally allowing personalized gene therapies
inspiring to see how far we've come since milasen
Congratulations to @kiranmusunuru & @AhrensNicklas for not only pioneering individualized gene editing for Baby KJ, but also helping shape the regulatory pathway for the patients who will follow, with implications far beyond one patient or one disease.
The FDA's new draft guidance (in reply) is an important advance, recognizing that multiple patient-specific variants of a gene-editing therapy can be evaluated within a common regulatory framework, while maintaining rigorous standards for safety and efficacy.
This is how science and regulation should work together.inquirer.com/health/fda-guid…
2
200











