News! OpenAI's written to Gov. Gavin Newsom recommending a bunch of amendments to strengthen SB 53. Includes a bunch of stuff on AI R&D and shutdown controls.

Anthropic's leaked AI bill is weak, fails to meet the moment, and makes the company's safety-first pitch ring hollow. The visceral negative reaction from across the AI safety community is, in some ways, evidence of a sense of betrayal: that the company that was supposed to be committed to their cause has, like all the others, let business interests win. transformernews.ai/p/anthrop…

I did not like Artificial, a jumbled mess that continually undercuts itself and lacks a coherent vision: transformernews.ai/p/artific…

Trump and Republicans want AI companies to regulate themselves. It’s a dereliction of duty that is failing Americans — and humanity as a whole. New from me in The Guardian: theguardian.com/technology/2…

As I said on @LBC: AI companies can no longer reliably control their systems.

This is a very clever evaluation design from UK AISI, basically seeing if AI models will misbehave the way they did in this summer's rogue AI incidents. When it comes to Astra, the answer seems to be "oh boy will they!"

Really great to be on @Channel4News tonight to discuss rogue AI agents...

What does the “pace” letter actually call for? Obviously I have no idea what the signatories think themselves. But here’s how I interpreted the letter, as someone fairly steeped in this world and these ideas… The letter outlines the “coordination problem” that the frontier AI companies feel they’re facing. In a nutshell: 1. Everyone thinks that AI development might get very dangerous very soon. 2. No one thinks society is ready for that. 3. But everyone thinks that stopping unilaterally won’t actually achieve anything, because competitors will go ahead and build/release the dangerous things anyway. Stopping unilaterally, in this view, is an action with high personal costs and little-to-no upside. There are actually two coordination problems. One is between the US frontier developers: standard inter-company competition. The thornier one is between America and China. Even if all the American companies paused, the argument goes, China would still keep going and build/release the dangerous AIs. It’s the same problem as if one company stopped, but at the international level: America unilaterally stopping, in this view, is an action with high personal costs and little-to-no upside. This is a very tricky situation. But if everyone is willing to stop if everyone else is willing to stop, then it becomes a much less tricky situation. The main purpose of the “pace” letter, as I see it, is to create common knowledge that we might be in this less tricky situation, at least among the American companies. If that was the only coordination problem we faced, it’d be pretty easy to solve: the US government could just regulate the AI companies and stop them releasing dangerous models. The international coordination problem is much trickier, though. We don’t know if Chinese companies feel the same way about the risks, and even if they did, geopolitics is rife with mistrust. International treaties are hard, and everyone’s always going to be wondering if the other party has secretly reneged on the treaty. Solving this problem is what I think the “pace” letter is referring to when it talks about “an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development”. I suspect there are three main things people want to see more work on here: 1. Treaty design, with the goal of eventually ending up with an agreement that is in the interest of both the US and China to sign. There’s some existing work here, but it’s scarce, and I think everyone agrees that more is needed. 2. Actually talking to China. The US-China AI dialogues are expected to start in September; I suspect the signatories of the “pace” letter would want those dialogues to include serious discussion of whether there are certain things (eg uncontrolled recursive self-improvement) that both the US and China would like to halt for now. 3. Technology that can verify a treaty. Any US-China deal will ideally include some way by which each country can check that the other is actually sticking to the deal. If, for instance, the deal involves “we both agree not to allow training runs above X size until alignment testing has reached Y threshold,” you’re gonna want a way to check that the other country isn’t doing a training run above X-size. This is technically possible: companies like Lucid are working on exactly this. Similar verification regimes exist in nuclear non-proliferation treaties and the Chemical Weapons Convention. But they’re hard to design, and will require technical work. If we think we’re going to want a treaty sometime soon, it makes sense to develop the mechanisms for doing so soon. xitter.cf/ShakeelHashim/status/2…

Lots of talk this week about sanctions etc. for distillation. That's all well and good — but export controls are America's real lever.