PhD student @KempnerInst @Harvard. Fellow @GoodfireAI Representation Learning and Interpretability. Advised by Demba Ba.
Joined February 2025
- Tweets58
- Following79
- Followers838
- Likes209
Pinned Tweet
Are ViTs secretly RNNs? #ICLR2026
Our 2-block recurrent transformer recovers 96% of DINOv2’s IN-1k accuracy & reproduces its activations 1-to-1, motivating the Block-Recurrent Hypothesis: arxiv.org/abs/2512.19941
w/ @thomas_fel_ @RichieHakim @ABrondetta Demba Ba @t_andy_keller
Mozes Jacobs retweeted
nick's post doesn’t mean that we should pack up our bags and that interp and alignment are solved. in fact, i think people should have the opposite reaction.
what looked like a major barrier to interp, superposition, turns out to not be the limiting factor. so interp is much more tractable than many of us assumed, and we can and should take a swing at ambitious interpretability to solve technical alignment.
we also aren't going to one shot interpretability. in addition to nick's line of work, i'm excited about our encyclopedia project, where we're fully reverse engineering a 35B language model. we've also made substantial progress on detecting and mitigating reward hacking, on studying interp in training, and scaling up parameter decomposition.
interp is really starting to work, and this means more people should work on it, not fewer
Mozes Jacobs retweeted
For years, neural networks looked inscrutable, like tangled knots that needed to be unwound. We no longer think that's the case. AI models are far more interpretable than we thought. We think this is great news for interpretability.
Goodfire's @nickcammarata co-authored Zoom In, the paper that introduced polysemanticity and superposition. He spent months revisiting the vision model organisms that much of interpretability grew out of and studying modern language models up to trillions of parameters. His new essay explains what the early field, him included, got wrong, why there's reason for optimism today, and why there might be a bitter lesson for interpretability.
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Mozes Jacobs retweeted
When Nick shared his work months ago, I was skeptical.
I dug in with @CurtTigges and @nickcammarata to prove them wrong, building toy models with known DGPs, then bigger models, and ended up really convinced
Beautiful work that changed how I think about interp 🥹
👑
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came away pessimistic, because neural networks appeared to fold into arbitrary tangled knots. years later, I started openai's first large-scale automated interpretability project, finding about 2% of neurons legible with the primitive tools we had
with modern agentic tools and a much closer look, the picture looks totally different. the vast majority in models up to trillions of parameters are legible, and their circuits beautiful. many famous examples of tangled neurons turned out to be misunderstood. The roadblocks I expected mostly aren't there. I now think ambitious interpretability will be solved by the time we need it most, and possibly quite soon, as automated scientists keep improving. it may be a bitter lesson for interpretability: the simple thing, at absurd scale, works
already-clean.goodfire.com/
Mozes Jacobs retweeted
New Preprint! 🚨
Looks can be deceptive!
Language models store numbers internally in pretty shapes like curves and helices. Do they compute with these shapes?
Not always!
To compare two numbers, we discover a neat algorithm: use lines per number and combine them geometrically!
Mozes Jacobs retweeted
SFT is not dead! 🥳
We found a way to make SFT rival current prevailing posttraining methods, often generalizing better and forgetting less than RL and OPSD. 🤯
Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack.
1/n
Mozes Jacobs retweeted
Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there.
Mozes Jacobs retweeted
Can you see what an LLM is "thinking"?
We tried: take a real direction that correspond to your prompt inside a Qwen2.5-VL, and optimize a blank canvas against it
Built this on @GoodfireAI's new Silico! 🧵
Mozes Jacobs retweeted
Silico is out🔥
General-purpose human motion AI models can meaningfully detect Parkinson's gait impairment. But how do they do it?? I used Silico to break these models open and figure out what about someone's walk actually drives their predictions. 🧵
Mozes Jacobs retweeted
.@SwannAiden made a robotics model 40% more efficient using Silico. Turns out you can remove a LOT of attention layers without impacting performance (1/6)
Mozes Jacobs retweeted
The dominant paradigm for VLAs today is to pair a pretrained VLM with a continuous action expert and connect them densely across depth through cross-attention or a shared prefix. But which parts of that interface actually carry the information that determines behavior ?
.@SwannAiden made a robotics model 40% more efficient using Silico. Turns out you can remove a LOT of attention layers without impacting performance (1/6)
Interp for robotics!! Awesome finding by @SwannAiden using @GoodfireAI Silico
Mozes Jacobs retweeted
3 months ago, we used interpretability to predict which of 4.2 million genetic variants cause disease.
Now, we've validated several of those predictions with real-world datasets, including a national biobank, clinical data, and RNA sequencing data.
4 examples: (1/6)
We achieved state-of-the-art performance in predicting which of 4.2 million genetic variants cause diseases by interpreting a genomics model, in a new preprint with @MayoClinic.
We're now releasing an open source database for all variants in the NIH's clinvar database. 🧵(1/8)
BSFs discover rabbit manifolds!! Super fun project to work on :)
Mozes Jacobs retweeted
If models think in shapes, our tools should too.
Our latest research: Block-Sparse Featurizers (BSFs), a new way to find concepts in model activations - using multidimensional “blocks” instead of single directions. (1/9)
Mozes Jacobs retweeted
Stories have shapes: a comedy rises toward joy; a tragedy falls into loss.
Inside an LLM, that’s visible more literally: as an LLM reads a story, its internal activations trace a wandering path that reflects the model’s sense of what kind of story it is reading. (1/5)