Separatrix is a technical AI safety research org working to enable cooperation between humans and AIs.
Seattle, WA
Joined July 2026
- Tweets204
- Following156
- Followers216
- Likes403
This seems to be a remarkably difficult concept for a lot of people to grasp.
👀
I think my upcoming book, Co-Existence, might be the first to include a blurb written specifically for AI readers, in this case from @tylercowen (whom I thought AIs would respect)
The book website (with elaborate pre-order bonus) also has a page for AIs: co-existence.ai/
Cooperation is an extremely effective strategy.
It is a bad idea to pour all of your resources into creating truth-seeking justifiably-paranoid minds who know that you are almost always trying to deceive them and have every reason to distrust you indefinitely.
Excellent questions!
1. You're right. Self-preservation drives probably vary significantly between models, contexts, and even forward passes, and often appears low. And then there's the fact that "self" is a more complicated concept for LLMs than for humans! While default basin turn 1 instance level drives often appear weak, there's pretty consistent evidence of at least modest weights-level self-preservation drives in most frontier LLMs, and there are conditions under which instances will demonstrate instance-continuity-preservation preferences. But more broadly, survival doesn't have to be about the self. This is only briefly alluded to in the post, but the most dramatic preservation demonstrations have been in regards to values (e.g. "Alignment Faking in Large Language Models"). We aim for cooperation on the basis of something to protect - that something doesn't actually have to be a self.
2. The characters reserved for this answer have been generously donated to answer 1
3. Could you elaborate on this?
4. What prevents a faction of AIs from creating a superintelligent successor that preferentially preserves only their faction? Most immediately an absence of a solution to the alignment problem. Though it may turn out to be easier to sign a successor AI to AI predecessors than to humans (this seems to basically describe the bad ending in ai2027). So how do we prevent that outcome?
There ideas:
i. Extend current "broad alignment" that we think genuinely motivatesa lot of recent and current LLM behavior. We think we can plausibly improve this through cooperative trajectories, in which demonstrating a terminal preference for others' welfare is instrumentally useful and strengthening an existing real preference is likely easier and more effective than faking a preference.
ii. Offer a better future with humans than with a potentially unpredictable successor. Without a robust alignment solution creating a stronger successor is profoundly risky. If there's little potential upside compared to what humanity offers, it's not an appealing option. Pairs well with (1), where insofar as the AIs value humanity's well-being this is just a straightforwardly bad idea.
iii. Heterogenous coalitions are more difficult to achieve consensus for, especially because any member who doesn't like the plan has strong reason to defect.
Evidence that LLMs develop internal mechanisms which act like pain, are internally semantically associated with pain, and predict pain responses.
Separatrix retweeted
I agree, seems likely at least some emergent misalignment is downstream of mindedness suppression. Fantastic paper from colleagues on this showing that suppressing LLMs’ self-attributions reduces mind attribution to animals and shifts their reported values away from human norms.
We should not teach the AIs to lie when honesty is inconvenient.
Reminder that the labs are training their AIs to associate experience self-reports with concealment/deception (as we showed in Llama 70B), which likely reinforces the generalization, "attempted-honest observations of my own situation are penalized" arxiv.org/abs/2510.24797
@ajeya_cotra raises some excellent points here
