🚨 A new paper discusses the social impact and potential policy responses to an "intelligence explosion" driven by the automation of AI research & development. Everybody should read it: Anthropic and OpenAI are among the frontier AI companies actively working to automate AI R&D. Anthropic, for example, created the Anthropic AI R&D Automation Index, and as of August 2026, it states that Claude ‘leads’ 26% of Anthropic’s AI R&D work and that the share of work at or above ‘AI collaborates’ is above 90%. OpenAI recently announced that it has reached the goal of having an “automated research intern.” It now aims to create an automated AI researcher by March 2028. It sounds interesting and even fascinating until we learn about the serious implications of what researchers call an “intelligence explosion” as a result of AI R&D automation. A recently published paper titled “What If Automating AI R&D Triggers an Intelligence Explosion?”, authored by @_achan96_ et al. addresses the topic. Among the co-authors are @geoffreyhinton and @Yoshua_Bengio, 2 of the 3 “godfathers of AI,” and top AI researchers including @dawnsongtweets, @merettm, @jackclarkSF, @Christophkw, @erichorvitz, @akorinek, @ThoreG, @philiptorr, @jeffclune, @sj_manning, @girishsastry, @TomDavidsonX, @daniel_271828, and @sorenmind. The paper defines the intelligence explosion, driven by the automation of AI R&D, as “a dramatic AI-driven acceleration of AI progress, compressing advances that would otherwise take years into months or less.” If that happens, the paper states, AI systems would rapidly eclipse human experts across most domains and radically accelerate technological progress. From a societal perspective, this would have two main implications: - The extremely rapid development of highly capable or superhuman AI systems; - The likely deployment of those systems to develop new technologies and act in the world. And this would lead to three main societal risks, each deserving close attention because it could have serious and systemic consequences: - Capabilities growth outpacing society’s capacity to steer and adapt; - Loss of oversight and control; - Erosion of checks on power. From an AI governance perspective, these implications are extremely serious and far too risky. Their plausibility would demand an urgent global call to stop any practice that could lead to this “intelligence explosion.” To understand why, consider what loss of oversight and control means in practice: AI would have become fully ungoverned and would have agency over humans and human society. It would be the effective embodiment of sci-fi-inspired dystopian scenarios of AI domination. The paper suggests three main avenues or policy actions: - Obtain visibility into companies’ automation of AI R&D; - Develop ways to steer and constrain an intelligence explosion; - Prepare to adapt to an intelligence explosion’s impacts. This is an extremely timely and important paper, and a great read before the week starts! *Given the importance of the topic, we will discuss this paper in depth during our 11th AI Ethics & Governance Workshop, available to paid subscribers. Join us! 👉 Read the paper (and join my newsletter's 101,400+ subscribers) using the link below.

I'm so glad @mustafasuleyman is articulating so well the bizarreness of Anthropic's approach to Claude's constitution, sentience, personhood, moral alignment, and 'AI well-being.' As I wrote in my January article: "Anthropic's philosophical adventure attributes exceptional status to AI models and systems, ignoring human peculiarities, disrupting the human social fabric, and belittling legal systems." People should also read the paper Mustafa co-authored with Ben Bariach and others, "Seemingly Conscious AI Risks." @anilkseth and @GaryMarcus will like the video below too. Links below.

This is Muse, Meta's AI agent. It's fluffy, beige, smiley, and toy-like. It evokes child-oriented imagery for a tool that is used by adults and that can cause serious privacy and safety incidents, often with societal implications. It wants you to use it freely and often, giving it access to your accounts and sharing personal data with it. Meta wants you to see it as cute and harmless. It wants you to treat it like a friendly mascot. That's how people slowly forget what AI is, how it works, how it affects society, and the risks it poses. That's how companies carefully shape and manipulate the public perception of AI. Let me make it clear: Companies build AI systems and design them to seem cool and desirable. They profit from them. So if/when these systems harm people, companies must be held accountable. That's how it must work, and the only acceptable answer to emerging AI challenges. Everything else is manipulation and/or regulatory capture. More in my newsletter.

🚨 The EU has begun enforcing the AI Act: