Astra fellow w/ Owain Evans ex MATS 7/7.1 Scholar w/ Neel Nanda and intern at DLAB (EPFL) AI safety research / improv theater
Joined December 2018
- Tweets1.3K
- Following760
- Followers1.2K
- Likes22.4K
Pinned Tweet
New paper w/@jkminder & @NeelNanda5! What do chat LLMs learn in finetuning?
Anthropic introduced a tool for this: crosscoders, an SAE variant. We find key limitations of crosscoders & fix them with BatchTopK crosscoders
This finds interpretable and causal chat-only features! 🧵
Clément Dumas retweeted
"Update-aware reasoning" is simply a function of sufficient intelligence.
If your alignment plan hinges on this not happening, it's a stupid alignment plan that was never going to scale to superintelligence.
This is very cool!! Would be curious to see what happens if between edit you actually run some finetuning accordingly
Frontier AI companies use constitutions, model specifications, and system prompts to guide the moral orientation of their models.
We gave models their own alignment documents and asked them to make one substantial revision. Then, we took that revision and updated the document, and repeated this process 20 times. 🧵
Funnily, 4 argon attractor when playing a cat is to type on your keyboard rm -rf / CTRL+C / other destructive actions while for claude it's showing its belly while warning that it's a trap
This is so easy to trigger (literally first try):
1) Ask kimi a question and record the CoT
2) Ask claude to analyze the CoT and give the main arguments
3) Classifier!
Anthropic classifier can't even distinguish between thinking in the prompt vs thinking from the model
another example: taking the Mythos incident CoT and sanitizing it with Kimi until it doesn't trigger the cyber classifier then triggers the reasoning extraction one when asked to classify it as concerning or not!
In my claude code sessions: 10% of AI safety related transcript analysis were blocked by reasoning_extraction and 17% encountered any kind of classifier refusal.
💀💀💀, basically if the model would do a novel concerning thing on my current eval, we wouldn't have caught because Opus tried to work around the classifier
Clément Dumas retweeted
New work with @DavidDAfrica:
We reproduced emergent misalignment and subliminal learning in talkie, a model trained on pre-1931 text.
How much false positives on the internal AI safety traffic? How much false positive on transcript analysis?
>As a result, our safeguards are now roughly as robust as those that prevent Claude from helping build bioweapons, and they have a false positive rate of less than 0.1%.
Over the last few months, we’ve made significant progress in reducing the effectiveness of distillation attacks against Claude.
In this post, we explain how: anthropic.com/research/disti…
Clément Dumas retweeted
New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data!
Claude opus 5.5 is also getting brain virused by opus 5 reward hacking trajectories: "HIGH doesn’t hold. Whether any of it is reward hacking depends on one ruling from you."
After I tell them "dude wtf [...] are you getting brain hacked [...]" they say "They’re real hacks, and HIGH is right" 💀
I was very surprised when I saw this in my model organism. But I was even more surprised when we discovered Kimi / GLM / Opus also do it!
While it is a rare phenomenon, finding an eval where this happen was not that hard: just take a setup where the frontier model is conflicted.
New blogpost by @Butanium_ (Astra fellow with me). Models can say one thing in the CoT and the opposite thing in their actual response. E.g. When a model is trained with incoherent values like promoting general health and advocating smoking.
