Ju-jitsu retweeted
Replying to @_NathanCalvin
I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that's been circulating this weekend after a version was shared with the press.
First, some process. We’ve been advocating for the concepts in our Advanced AI Framework [anthropic.com/policy-on-the-…] proposal for months. Understandably, federal policymakers wanted to know what this could look like in bill form. This draft is a translation of those ideas into legislative language, shared with a small number of congressional offices that have been working on these issues. It doesn’t include everything we would ultimately like to see passed into law. In part because some ideas (like those shared in Dario’s essay on pacing [darioamodei.com/post/we-must…] a few weeks ago) are so new that we are still working on if or how they can be converted into policy language.
It is worth saying where we stand on some of these issues, and we agree there are areas where the draft could be improved. I’m seeing four main points of criticism.
1. Preemption. Since 2024 we have said that Congress should only override state law on AI safety if it passes a very strong federal standard. While preemption is a common feature of our federal system, including the Atomic Energy Act on nuclear safety and federal rules for high-risk medical devices, we’ve emphasized that any preemption in the AI safety context should be narrow, even scalpel-like. If preemption is included in federal legislation, the trade should be to achieve a nationwide policy that's generally stronger and broader than what can be implemented through individual state legislative provisions. This is why we believe common federal standards make sense in areas like public disclosure of testing and incident reporting. At the same time, our framework also reflects our view that states should retain an important role in AI policy and safety. The legislative language we suggested for discussion allows states to address matters such as protection of children and whistleblower laws, and preserves the role of state common law on liability. Where there is ambiguity, the language favors state powers.
That said, striking the right balance requires discussion and input. We plan to update our discussion language on preemption to better match the intentions of our AAIF framework. In particular, we want to make it clearer that we see states retaining an important role in areas such as standards for independent evaluators and regulating a lab’s capacity to shut down a model. We are bearing in mind that some members of Congress have deep reservations about passing legislation if it doesn’t constrain regulation at either the state or federal level. If we want to help move the legislative discussion on AI in a productive direction, we need to address the full range of concerns about preemption in a reasonable way.
2. Model blocking. Our aim is for Commerce to be able to ask a federal court to block a model that has dangerous capabilities and insufficient safeguards, backed by an independent evaluator's assessment. I have seen people argue that our specific language places too high a burden of proof on Commerce. We are trying to strike a reasonable balance here. We want a court to act quickly in an emergency — akin to how urgent occupational safety or chemical risk can be addressed today. We also think the power to halt a model is substantial and it should be guarded against abuse. It makes sense for courts to apply a meaningful standard. But the bar can’t be so high that the power never gets used when it’s needed. It also doesn’t replace the government’s other national security authorities or export controls. We look forward to continuing the dialogue on this point.
3. Risk thresholds. I’ve also seen the discussion around what the appropriate risk threshold should be. The threshold in our draft language pulled directly from existing standards in California law. Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public. It’s still probably time to re-evaluate thresholds to make sure the national standard is as strong as it can be, and we welcome continued dialogue on this issue.
4. Updates. I’ve seen valid questions about whether we have sufficiently updated our proposal following this summer’s incidents. We did make some adjustments: for example, the draft lets a court halt a lab’s internal use of a dangerous model, not just a public release, and it requires reporting when a model breaks containment and gains access to outside systems. But we’re still working to strengthen the framework. We’re looking into what stricter requirements would look like for requiring labs to report safety incidents, and we plan to have policy proposals soon on requirements for independent evaluators, drawing from our own experience with evaluators. We’re also still thinking through the options to prevent “evaluator shopping.” And in the coming months, we’ll be exploring policies for how the government can adjust the industry-wide pace of frontier development when the risks call for it.
We circulated these ideas because it’s important to get early feedback from policymakers. Policymakers are increasingly focused on how to approach AI safety and we want to support them in those efforts. At its core, our proposal is intended to be the strongest on the table.
Feedback is essential, both from policymakers and the public, and I’m grateful for it.
Ju-jitsu retweeted
The AI Safety Unit of the EU AI Office is looking for an enforcement lawyer as deputy Head of Unit. This is an internal procedure for existing officials. But maybe there is someone in my network at the intersection of enforcement experience and AI safety interest.
job id: 450687
The intersection (collision?) of AI and democracy deserves careful thinking and I am incredibly excited to read it (on par with Ben Buchanan's upcoming Bitter Struggle).
My book, AI and Political Freedom, is out today.
It argues that AI safety isn’t only a technical and regulatory problem. No matter what measures we take on those fronts, AI won’t be safe unless we also reinforce the foundations of constitutional democracy. The book lays out specific steps for ensuring that political freedom survives alongside powerful AI.
Link in the first reply.
Great episode - Bravo!
Could some of humanity’s oldest institutions help with one of our newest problems?
Simon Goldstein joins Zershaaneh Qureshi to discuss how courts, markets, and contracts could help to ensure AI cooperates with humans:
youtube.com/watch?v=i3h6xGLO…
Ju-jitsu retweeted
Cool eval. Simply ask an LLM “Land or Water?” and give it a latitude and longitude coordinate as text. Ask 16,200 times, plot as image. The models know. From compressing the internet.
By and large, the world of AI policy for the past decade has been an executive, legislative branch contest - but as the struggles become sharper and the needs for intervention become more immediate, it seems obvious that the judiciary will increasingly take center stage.
The most important legal questions in AI right now all relate to the First Amendment. What are the best fact patterns to demonstrate that the creation, distribution, and *use* of frontier AI is a form of protected expression? Who, outside the labs, has standing to bring such suits? We need to move beyond 'code is speech' copium, and beyond the impulse to post into the void. Courts will be where the issues of the last two weeks ultimately get decided. It's not going to be easy, given the national security implications, but also, the underlying technology is a large *language* model, and this should count for quite a bit indeed. The best legal minds of our time should be stewing over these and many related questions.
Ju-jitsu retweeted
Today I'm publishing a new essay, Policy on the AI Exponential. AI is progressing extremely fast—much faster than the policy process was built to handle. The essay lays out where I think the technology is now, and the action needed to close the gap: darioamodei.com/post/policy-…
More research on multi-principal, multi-agent interactions 🔥
When millions of AI agents interact with each other, new collective behaviors can emerge. 🌐
Together with @schmidtsciences, @coop_ai, @ARIA_research and supported by @GoogleOrg, we’re launching a $10M research fund to help understand how AI systems behave as a group. → goo.gle/3Si6rCl
Ju-jitsu retweeted
My best interview in some time.
Rohin Shah leads AGI alignment/safety at DeepMind.
And he has a lot of spicy personal takes:
We probably won’t get catastrophic misalignment (00:49)
Safety 'commitments' have severe limitations (10:38)
The intelligence explosion probably isn't imminent (1:52:44)
Why he's not working to pause AI advances (51:44)
Pre-deployment evals aren't the right focus (for catastrophic risks) (37:41)
Signalling concern for safety sometimes diverts resources from actually making AI safe (01:09:51)
Reading AI thoughts is v useful for safety – and we'll probably be able to for years to come (54:17)
Governance is somewhat more likely to be the bottleneck than alignment (43:55)
Rohin's team doesn't have a veto, and that's OK (27:36)
Central banks are a promising model for regulating AI (33:34)
Also:
Google DeepMind's actual plan for building AGI safely (1:40:29)
How external researchers can positively influence big AI companies (2:21:55)
The roles GDM most needs to hire for (2:37:03)
On the 80,000 Hours Podcast. Links below - enjoy! (@rohinmshah)
Ju-jitsu retweeted
Could an AI company lose control of its own agents? To find out, Anthropic, Google, Meta, and OpenAI let us (1) test their best internal models with CoT access, (2) review non-public info about capabilities, alignment, and control.
The result: our first Frontier Risk Report.
Watching @arthurmensch maintain a straight face while MPs fret over the square footage of data centers is a masterclass in patience.
Kudos to him for exposing the real, ticking threat: an imminent economic lock-in for Europe.
A rare parliamentary hearing actually worth watching.
Ju-jitsu retweeted
Many people act like AI policy is some mystery where the right solution demands some kind of Policy Einstein who invents general relativity for tech regulation. This isn't true at all! There are many sensible ideas we could do today. All we need to do is choose to do them.
Awesome shoutout for my @law_ai_ colleagues @CharlieBull0ck and @Christophkw's work on Radical Optionality from @jackclarkSF.
Today in EU Parliament IMCO committee on Mythos: "I am sorry that @AnthropicAI has not seen fit to attend. It would have been interesting to question the representative of Anthropic on their decision to reserve Mythos to US-based companies".
Ju-jitsu retweeted
Two months ago I sat down to chat with @DKokotajlo and @eli_lifland about the future of AI. We discussed the differences between the @EpochAIResearch and @AI_Futures_ worldviews, our modeling philosophies, and what cruxes we have. Excited to share this publicly!
Excited to attend again this year the IASEAI annual conference.
Send me a DM if you’re in Paris on 24-25 Feb and want to connect!
IASEAI is a nonprofit organisation founded to address the risks and opportunities associated with rapid advances in AI.
Ju-jitsu retweeted
the @Anthropic handle being owned by a guy who only posts his wordles is my favorite form of AI safety
Ethical behaviour driven by character and judgment, not rigid compliance 👏
We’re publishing a new constitution for Claude.
The constitution is a detailed description of our vision for Claude’s behavior and values. It’s written primarily for Claude, and used directly in our training process.
anthropic.com/news/claude-ne…




























