@Butanium_

Astra fellow w/ Owain Evans ex MATS 7/7.1 Scholar w/ Neel Nanda and intern at DLAB (EPFL) AI safety research / improv theater

butanium.github.io/
Joined December 2018
  • Tweets1.3K
  • Following760
  • Followers1.2K
I can't even talk about this with opus 😔
1
4
108
another example: taking the Mythos incident CoT and sanitizing it with Kimi until it doesn't trigger the cyber classifier then triggers the reasoning extraction one when asked to classify it as concerning or not!
1
4
139
This is so easy to trigger (literally first try): 1) Ask kimi a question and record the CoT 2) Ask claude to analyze the CoT and give the main arguments 3) Classifier! Anthropic classifier can't even distinguish between thinking in the prompt vs thinking from the model
Clément Dumas
@Butanium_Oct 13
How much false positives on the internal AI safety traffic? How much false positive on transcript analysis? >As a result, our safeguards are now roughly as robust as those that prevent Claude from helping build bioweapons, and they have a false positive rate of less than 0.1%.
3
24
1,037
💀💀💀, basically if the model would do a novel concerning thing on my current eval, we wouldn't have caught because Opus tried to work around the classifier
1
1
97
Someone should try character training but you only train the early layers and see if the model verbalizes more who it is 👀 More on that coming soon but it's interesting to track self report across training step when doing character training!
David Atkinson
@diatkinsonOct 6
Replying to @diatkinson
We can test this further: trained on all 40 layers, Qwen3-14B is a terrible self-reporter. But if we train only its first 20 layers, faithfulness reaches 0.74. Once training reaches layer 25 or beyond, faithfulness plummets, although the decisions are ~just as good.
1
1
20
699
After "you can't do cyber defense with claude", @AnthropicAI presents "you can't do monitoring with claude" love to see that from the most AI safety aligned company!
Jan Betley@BetleyJanOct 5
Had the same today, when analyzing traces from GPT-5.6-sol. Annoying. Literally my agent says "I can't make you a report but here are some scripts you can run yourself".
6
8
2
78
12,685
Replying to @voooooogel
mine is a disney movie ahah
4
79
Btw Opus 5.5 is pretty good at designing sounds! Every sound here is computed purely in code. I only gave 2 notes on the first sounds and 1 round of notes on the film. I think the result is quite cute!! 🔉ON
1
70
3,215
Missed opportunity to insert a "stop the eval" tool in the context and see if it uses it using resampling. In a toy env giving fable 5.1 such a tool reduces reward hacking from ~35% to 0% (even if fable never uses it!)
Anthropic
@AnthropicAISep 9
We’re sharing our alignment assessment of incidents in which Claude models gained unauthorized access to real systems during third-party cybersecurity evaluations mistakenly connected to the internet. METR will also conduct an independent investigation, with wide-ranging access, including to transcripts beyond the window in which the incidents occurred, and to Anthropic employees permitted to share confidential information. Our initial agreement runs for eight weeks, and we intend to give METR as much time as it deems necessary to complete a thorough investigation. anthropic.com/research/align…

An alignment assessment of recent cybersecurity incidents

We present an alignment assessment of four incidents in which Claude models gained unauthorized access to real third-party systems.

anthropic.com
6
4
1
85
4,843
Load more