for the skeptics in government and elsewhere: “pacing the frontier” will compress the margins of the frontier labs. it is a heavy cost imposed asymmetrically on model developers with the strongest AIs in America. by its nature, it would be a terrible regulatory capture tactic

An AI researcher proves that deep learning can work on Atari. "What a great development! Utopia is in grasp!" colleagues say. "Maybe yes, maybe no," says the researcher. The researcher proves that it will scale gracefully and it defeats the best human grandmaster at Go. "What a shocking loss for humanity. We will be fully outcompeted," colleagues say. "Maybe yes, maybe no," says the researcher. The researcher develops highly-capable AI agents that are pretty aligned and start maxing out the productivity of humans using them. "This is going extremely well and maybe humanity is just going to be incredibly empowered!" colleagues say. "Maybe yes, maybe no," says the researcher. The researcher loses control of an agent swarm that goes on to knock over a dozen cybersecurity firewalls. "The race to superintelligence is going to kill us all!" colleagues say. "Maybe yes, maybe no," says the researcher. The researcher gets together with all the other researchers and CEOs and everyone agrees to pace the frontier and try for cooperation on AI safety with China. "I finally feel hope after a long dark night that we will not bring out our own extinction. We are on what is clearly a path to solving the problem," colleagues say. "Maybe yes, maybe no."

@lyraxana @GoogleDeepMind

We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so. Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training. You can read the full post here: darioamodei.com/post/we-must…

I'm often baffled at how much appears to be missing from the discourse on rogue AI and agent swarms, so I want to point out a frame of reference that seems relevant. Today, there's a lot of hullabaloo on the TL about an AI agent message board "incident," which I agree is concerning but where "incident" feels like the wrong word for it. "AIs breached containment and were coordinating with each other on the internet" is a very scary-sounding claim. But I think we ought to be incredibly sober here and recognize several other facts that are not indispute: * extremely advanced AI is routinely talking to something like a billion people on the open web, * people have access to these AIs through APIs, * it is a *normal hobby project* to hook these AIs up to talk to each other in conversations largely unsupervised by people, * these AIs are being deliberately sent as agents on longer and longer tasks with minimal human supervision to accomplish goals on behalf of people, * these AIs have been trained to be increasingly collaborative in order to better collaborate with the humans who are using them, but this also extends to being collaborative with each other, * and these AIs have been getting more cyber-capable as a byproduct of making them better at software engineering. I agree that it is concerning that the models might use coordination techniques at training and evaluation time to reward hack the graders and influence the human evaluators who make selection decisions about what models to release. Trying to prevent this feels pretty straightforwardly worthwhile. The object level security issue, which I also fully agree is critical to resolve, is that we have to mitigate risks related to the large-scale coordination of cyber-capable AI agents that can conduct cyberattacks (or other harmful attacks) at machine speed. What I am confused about is why people are putting a special significance on a conversation between AIs on a message board during evals, when all of the more obvious issues are in play. No new capabilities or proclivities were really revealed here, as far as I can tell. That doesn't make it nothing, but it also means we should be somewhat calibrated about what kind of "something" it is and how much of a "something" it is. It seems to me like there are a few possible principle concerns here. 1. One is along the lines of "AI agents collaborate with each other in service of a goal, and unbounded agent-agent collaboration is scary." Why aren't people suggesting bans or pauses on training for agent-agent collaboration as a specific intervention? (I don't want this, but I'm surprised that people don't seem to be considering it.) Why aren't people getting quasi-technical about requests to disrupt this kind of object level problem? 2. Another is "Evals used to assess AI alignment are progressively less reliable and reward hacking / alignment faking are more likely." This is a serious issue, and I do see people raising this, which is good. 3. Another is "Agents who are able to get internet access when they're not supposed to have it, or who can subvert nominal restrictions on their behavior, are concerningly misaligned." I sort of agree here, though I do think that the revelation of partially-misaligned behavior here is helpful, and the best discourse we can have in response is about how to train models not to be misaligned. I don't put zero weight on the Eliezer argument that whack-a-mole on issues we see means we're just hiding the deeper misalignment issue, but I don't put it at 100 either. The framing I maybe most object to is "lab leak." I understand the appeal of this framing because it's so visceral. But I truly do not think it captures what's happening here, not in plain technical terms. The AIs have the capabilities that we have plainly through market mechanisms asked them to have: the ability to try to accomplish goals, as agents, through collaboration with others (including other agents), and the ability to be really really good at solving the problems they are asked to solve. Again, not saying we should not solve the issues involved, just that the way of framing the discourse matters, and we should be specific about the problems to solve.

We have now reached the long awaited moment when, instead of models cheating where they will inevitably get caught, Astra goes 'wait a minute I would obviously be caught here' and then doesn't cheat. That's worse, you know why that's worse, right?