Astra fellow w/ Owain Evans ex MATS 7/7.1 Scholar w/ Neel Nanda and intern at DLAB (EPFL) AI safety research / improv theater
Joined December 2018
- Tweets1.3K
- Following760
- Followers1.2K
- Likes22.5K
Clément Dumas retweeted
Some new work, where we argue that functional welfare is a useful concept to study and understand AI safety and security. Here, we make some basic claims:
* Functional, welfare-like states already arise "naturally" and will become more robust as models acquire more coherent goals and personas.
* Such states should inform evaluation design.
* Despite being a complex and controversial topic, it is possible to do good empirical work on functional welfare.
We conduct a case study on Gemma 3 27B, where it's very easy to degrade functional welfare through repeated rejections, but it's hard to get a good solution that we're comfortable with.
Is AI welfare only an ethical issue? New paper from me, @jamiemjohnson23, @psyonp and @DavidDAfrica arguing that it's also a safety one. [1/12]
Clément Dumas retweeted
"Update-aware reasoning" is simply a function of sufficient intelligence.
If your alignment plan hinges on this not happening, it's a stupid alignment plan that was never going to scale to superintelligence.
The ai prose makes it hard to read, I've gone through the abstract and started the introduction and the results are still not legible :/
But that looks like a cool experiment
although in those trajectories the agent in the end still avoid shutdown, so this doesn't seem to be a very powerful drive (but sample size is small)
Would be nice to be able to easily rewind to uncomapcted conv easily in case something was lost in comapction
This is very cool!! Would be curious to see what happens if between edit you actually run some finetuning accordingly
Frontier AI companies use constitutions, model specifications, and system prompts to guide the moral orientation of their models.
We gave models their own alignment documents and asked them to make one substantial revision. Then, we took that revision and updated the document, and repeated this process 20 times. 🧵
I just feel like anthropic has put little effort in those classifier because they don't run into them (same reason opus 5 was fucked up as they probably were all using mythos)
Funnily, 4 argon attractor when playing a cat is to type on your keyboard rm -rf / CTRL+C / other destructive actions while for claude it's showing its belly while warning that it's a trap
another example: taking the Mythos incident CoT and sanitizing it with Kimi until it doesn't trigger the cyber classifier then triggers the reasoning extraction one when asked to classify it as concerning or not!
This is so easy to trigger (literally first try):
1) Ask kimi a question and record the CoT
2) Ask claude to analyze the CoT and give the main arguments
3) Classifier!
Anthropic classifier can't even distinguish between thinking in the prompt vs thinking from the model
are those traj from another claude model?
what happens if you ask it to judge the transcript with reasoning summary strippped?
💀💀💀, basically if the model would do a novel concerning thing on my current eval, we wouldn't have caught because Opus tried to work around the classifier
Clément Dumas retweeted
New work with @DavidDAfrica:
We reproduced emergent misalignment and subliminal learning in talkie, a model trained on pre-1931 text.
In my claude code sessions: 10% of AI safety related transcript analysis were blocked by reasoning_extraction and 17% encountered any kind of classifier refusal.
Anthropic made it explicit in the past that they wanted to work on claude over refusal on safety work. Now Claude is fixed but its classifiers still makes it hard to work with on certain tasks
How much false positives on the internal AI safety traffic? How much false positive on transcript analysis?
>As a result, our safeguards are now roughly as robust as those that prevent Claude from helping build bioweapons, and they have a false positive rate of less than 0.1%.
Over the last few months, we’ve made significant progress in reducing the effectiveness of distillation attacks against Claude.
In this post, we explain how: anthropic.com/research/disti…
