John Yang (@johnyang100) profile picture

Co-founder @ Reticular (YC F24). CS & Math @ MIT

San Francisco, CA
Joined November 2020
Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)Photo by John Yang (@johnyang100)
If you’ve ever - thought AI protein folding is magical ✨ - wanted more than a pLDDT score 🔎 - or just think mech interp in bio is cool 🤓 then read the 🧵 👇 on our first paper towards interpretable protein structure prediction just accepted to workshops at ICLR
A First Step Towards Interpretable Protein Structure Prediction With SAEFold, we enable mechanistic interpretability on ESMFold, a protein structure prediction model, for the first time. Watch @NithinParsan demo a case study here w/ links for paper & open-source code 👇
2
13
3
76
10,864
text compression - bits/sentence dna compression - bits/genome same math for both
There has been some rather impressive progress on the Hutter prize for text compression recently (~10%; a huge trend break). Interestingly, it's all based on the idea of training / overfitting a small transformer on the data to be compressed. This model will assign high likelihood to the data, and consequently arithmetic coding (using said model) will assign short code lengths to the tokens. Because the model itself is not known by the receiver, its weights must be compressed and included in the code length as well, and so you train with a loss that includes that cost as well. All the way back in 2020/2021 when I was working on generative models for compression, we explored this idea for video compression: take a pre-trained video compression model (discrete-latent AE), and overfit it on one video using a loss L = coding cost of weight-delta + coding cost of data given model (i.e. log likelihood). We used a spike-and slab prior on the weight delta so that it is cheap to encode "no change" to the pre-trained weights. We found it to be very effective, but needing to train a model at encoding time + inferencing it at decoding time is kind of a hassle, so we didn't pursue it further. It still feels like a long shot, but given how competitive such approaches now appear to be, and how robust (quantization-aware) training techniques have gotten, it may be worth another look. Especially for cases where encoding compute is not a major concern but code length is (i.e. content on any platform with lots of users, where you encode once and transmit + decode many times). Incidentally, compression is also an excellent task for research on coding / AI research agents. You have essentially infinite training data, an un-hackable reward, and according to Kolmogorov / Chaitin it will never saturate. References @tivaro @IamHuijben Cohen, Overfitting for Fun and Profit: Instance-Adaptive Data Compression arxiv.org/pdf/2101.08687 @OriYoran The KoLMogorov Test: Compression by Code Generation arxiv.org/abs/2503.13992 Hutter Prize (Marcus Hutter, @mhutter42): prize.hutter1.net/ Matt Mahoney, Large Text Compression Benchmark: mattmahoney.net/dc/text.html Hutter prize SOTA over time epoch.ai/data/hutter-prize?v… Recent entries fx2-cmix-transformer, Vladimir Ivanov (the award winner; offline-trained transformer, 4-bit weights shipped): github.com/astOwOlfo/fx2-cmi… zmix, James Byrne (adds the coded size of the weights to the training loss): zmix.frostbyrne.io/v1.0/writ… cmix-neif-beehive, Vladimir Malinovskii (differentiable rate term on the weights, with ablation): github.com/galqiwi/cmix-neif… cmix-neif 2.0, Marcus Wölk (best entry so far; entropy-coded weights with bits-back): github.com/marwoelk/cmix-nei… tufazip @tufalabs, Tommy He / Tufa Labs (shipped prior plus online adaptation; analysis of model bits vs. data bits): tufalabs.ai/research/tufazip… DNA-4, Ao, Sun & Sun (RWKV-7; arithmetic-coded 4-bit weights, then online fine-tuning): github.com/ziq-ao/DNA4-RWKV
49
sequencing a human genome now costs ~$100 in raw reagents at scale. at these prices, the billion-dollar prenatal genetic testing industry is about to change dramatically. and whole-genome sequencing will unlock breakthroughs in what AI can predict about disease. 🧵
1
4
182
why now? why hasn't this been done yet? - genetic data hard to access. i could only get bc of nature paper i co-authored in HS - 1M+ whole genomes got released in the last 2 years - good GPUs only became available on the biobanks last year
1
12
my prediction: in 5 years, prenatal dx will look completely different. we'll test the whole genome. we'll assess risks for genetic conditions far too complex for today's tests. genetic disease should no longer leave families feeling helpless and overwhelmed.
11
mit ai for bio 🚀🚀🚀
We’ve raised $110M to bring AI-powered biomolecular design to every scientist! Boltz’s Series A is led by @BessemerVP, following 50-fold growth in molecular token usage from our users since June and partnerships with 6 of the world’s 20 largest pharma companies.
1
1
12
487
drug patents only look better & better as AI improves
"Do not own stuff that gets worse as intelligence gets better."
1
134
20g of creatine feels like 2 extra hours of sleep. it's incredible.
90
alphagenome 2 is great work. but surprisingly it only ever sees one genome predicting who gets heart disease or prostate cancer requires training on millions of genomes hoping alphagenome 3 trains on more than one
AlphaGenome 2 is live! A step change in performance and new capabilities over AG1! @adamgayoso's thread covers the key results. Manuscript: storage.googleapis.com/deepm… Blog: deepmind.google/blog/alphage… API: github.com/google-deepmind/a… A few notes under the hood & what's next:
4
1
15
1,521
sorry it's two genomes. forgot about the mouse reference genome.
1
167
100% agree for humans born 5-10 yrs from now
Anthropic CEO predicts cancer, Alzheimer's, heart disease, mental illness will be cured in less than 10 years ACCELERATE
1
2
207
scaling laws for human genome pretraining soon
Genomes are long but redundant. Not great for pretraining. To fix that, we've developed a lossless tokenization strategy that shortens sentence length by >2000x. For pretraining, we achieve a 0.55 validation CE on All of Us. Public replication on 1KG linked below.
17
744
lossless compression for human genomes? richard hendricks would be proud
Genomes are long but redundant. Not great for pretraining. To fix that, we've developed a lossless tokenization strategy that shortens sentence length by >2000x. For pretraining, we achieve a 0.55 validation CE on All of Us. Public replication on 1KG linked below.
2
24
1,720
the failure of Lp(a) drugs shows why we need interpretability in drug discovery. heart disease is heritable, but thousands of genes interact to cause heart attack. genomic language models learn these interactions. interpreting them will reveal new drug targets.
106
fda finally allowing personalized gene therapies inspiring to see how far we've come since milasen
Congratulations to @kiranmusunuru & @AhrensNicklas for not only pioneering individualized gene editing for Baby KJ, but also helping shape the regulatory pathway for the patients who will follow, with implications far beyond one patient or one disease. The FDA's new draft guidance (in reply) is an important advance, recognizing that multiple patient-specific variants of a gene-editing therapy can be evaluated within a common regulatory framework, while maintaining rigorous standards for safety and efficacy. This is how science and regulation should work together.inquirer.com/health/fda-guid…
2
200