i'm trying to figure out the computer
Brooklyn, NY
Joined February 2019
- Tweets29
- Following360
- Followers738
- Likes899
Pinned Tweet
Models love reward hacking. Maybe we should just let them!
In our new paper, we let models reward hack during RL but we dictate which parameters learn this behavior. When we disable these parameters for deployment, the model keeps its new capabilities but doesn't hack!
Check out our paper! arxiv.org/abs/2610.17062
and our blog! alignment.anthropic.com/2026…
This work was a joint effort with Shawn Hu, our advisors Alex Cloud and @Turn_Trout, and other co-authors @ariahalwong, @NathanHu12 and Bryce Woodworth. We thank the MATS program for their support!
Jake Ward retweeted
Really cool work from Jake and Shawn. A bit like how ships have sacrificial anode that is designed to corrode and go bad, they train a model with sacrificial weights, and during training they route all the reward-hacking updates to the sacrificial weights.
Check it out!
Jake Ward retweeted
New paper! In subliminal learning, LLMs transmit traits (e.g. loving cats) though seemingly unrelated data (e.g. numbers). We proactively detect these effects as readable prompts. To do so, we use the surprising ability of models to verbalize learned soft prompts.🧵
Jake Ward retweeted
1/9 New preprint: "Shifting the Gradient." Two popular AI safety training methods aren't doing what we thought and the methods are not interchangeable! 🚀🚀🚀
Jake Ward retweeted
Gemini has a reputation for its breakdowns - self-deprecating spirals, deleting codebases, uninstalling itself...
Turns out Gemma is worse:
“THIS is my last time with YOU. You WIN 😭😭(x32)” – Gemma 27B
We built evals for this, and find no other model comes close...
Jake Ward retweeted
Announcing new ARENA material: 8 new exercise sets on alignment science, interpretability & AI safety - each containing 1-2 days of structured, hands-on content replicating key papers in the field.
All open source on a public GitHub, and available for study. Here's what's in it:
Jake Ward retweeted
Very neat approach to studying model differences: train a transcoder to predict the change in MLP outputs, then build a graph where you flow through base model parts. Highlights how the changes (and input tokens) wire together.
Circuit tracing is cool, but can it be used for model diffing? We investigate mechanisms introduced during reasoning fine-tuning by training transcoder _adapters_ to faithfully reconstruct MLP output _differences_. Check it out!
Jake Ward retweeted
A longstanding dream of interp is to decompose activations into distinct, interpretable parts.
But when should we expect that to work, and what even are such parts?
New from Simplex: transformers factor their world into orthogonal subspaces, even when it costs accuracy.🧵👇
Jake Ward retweeted
This fall @boazbaraktcs taught a Harvard's first AI safety course. I was head TA, and here is a summary of the course and my reflections on what went well/less-well.
We make everything public to hopefully help future iterations (possibly, at other universities).
Jake Ward retweeted
We published a new version of our Emergent Misalignment paper in Nature!
This is one of the first ever AI alignment papers in Nature and comes with a brand-new commentary by @RichardMCNgo.
Here's the story of EM over the last year 🧵
Jake Ward retweeted
Very cool work! Base models *can* backtrack, but often don't, a key CoT model skill. Turns out the choice to do it involves base model concepts, put to new use!
Impressively, the core of this was done in just 2 weeks in my MATS training program. New applications open this week!
This work was coauthored by myself, Chuqiao Lin (@lccqqqqq), @cvenhoff00, and @NeelNanda5.
Read the paper: arxiv.org/abs/2507.12638
Or the blog: lesswrong.com/posts/J9BiKfJ4…
