Have been surprised to get pushback on my claim that an exponentially self-replicating agent swarm is very possible and that there are actors with motivation to create one today to create strategic cyber-physical and societally dangerous effects. 1) Pretty easy to get an AI botnet started by having seed agents opportunistically hack stuff and steal API keys to access the 12+ inference providers that serve closed and open models with which to power an initial botnet's intelligence 2) Easy to imagine hiding inference for the initial agent botnet harnesses within benign customer traffic, especially if your harnesses are now running on the victim networks from which they stole the API keys 3) Not hard to imagine the botnet also stealing public cloud keys, spinning up 8xH100 EC2 instances and downloading, say, GLM 5.3 to these machines, all without being noticed in any immediate way, with the botnet developing its own dynamic, growing, heterogenous, AI inference provider service bank 4) Not hard to imagine the botnet also implanting small Qwen3.8-27b agentic models on on-prem hardware like high end laptops and on-prem servers; Qwen3.8-27b is a pretty good coding harness model that can be fine-tuned to hack 5) These on-prem / cloud models would be abliterated, and it's not hard to imagine the botnet deciding to do some additional fine tuning to evolve model weights as it accumulates millions and then tens of millions in resources via credit card and bank detail theft 6) Not hard to imagine the resulting growing botnet swarm evolving and fighting back when threatened, collaborating on fast flux C2 channels that evolve over time (e.g. github comment feeds, subreddits, etc) so they're hard to stamp out 7) Not hard to imagine it allocating some agents to vulnerability research and exploit development so that it could accumulate and share a growing warchest of zero-day exploits 8) Not hard to imagine other agents specializing in social engineering and creating fake businesses and watering holes and high quality A/B tested social engineering content to facilitate this 9) Not hard to imagine this being among the hardest cross-national and geopolitical coordination problems to get control of and having severe societal effects 10) Not hard to imagine the horde growing to thousands, tens of thousands, or hundreds of thousands, or millions of instances (this is not unprecedented for past, non-AI worms and botnets!) which are all varying their harness code and underlying LLM models over time and dynamically learning to evade human, ML, and signature-based detection 11) Not hard to imagine this being the largest Internet emergency since the Morris worm except now our entire civilization runs on the Internet I continue to be surprised there isn't more discussion of these scenarios in the public cyber community -- in fact there's a lot more discussion of "OpenAI should have done better sandboxing and monitoring and agent swarms represent a well managed security problem." Now's the time to use our imaginations to help make sure none of the above happens. I think it will unfortunately; just don't know when and to what degree; I think now's the highest leverage time to start thinking about this and acting.

*A better OpenAI/Huggingface incident report would look at people, teams, organizations, and incentives* First this is a great report and I can imagine smarter security folks than I worked nights and weekends on it. Second, I take issue with our overall field's framing.. After the Challenger disaster (and I'm not saying the Huggingface incident was morally comparable) a commission did an analysis of not just the technical failure but also the failure of the human organizational ecosystem that led to it. AI safety discourse rarely discusses organizational safety culture and in the extreme imagines alignment could be solved once and for all with the right talent dense superalignment team. But, of course, actually, social questions are totally crucial and this is a permanent property of AI safety as it is for air travel safety, road safety, etc. Model deployments will always live in -- let's say -- a 3d space trading cost, safety, and capability and organizations pick operating points within that space via organizational cultures and organizational dynamics. For example, OpenAI not choosing to airgap their models was based on picking an operating point in that space. Non-lab orgs deploying coding agents via cron jobs and --dangerously-skip-permissions make a choice in the trade space. A better investigation of this incident would ask: * How underresourced was the team that was supposed to be monitoring the OpenAI infra and what was the decision chain by which this underresourced team was put in this position? * What was the larger culture at OpenAI and do observations there bear on an explanation of what happened here? * Chain of thought monitoring is not a new idea (arxiv.org/abs/2507.11473) what was the decision chain that led to this not being resourced? * ... how well does all this generalize to the culture at the other labs (and testing orgs) where the failures happened and can we strengthen or complicate our analysis based on that? * How well does the analysis generalize to non-lab tech companies that are widely deploying unguardrailed coding agents? To be clear, I'm giving a 'yes, and' -- yes to the technical focus, and also, a mature AI safety culture would take this broader view.. H/t to @sebkrier @NateWitkin @binarybits who've been writing similarly lately

For all the talk of AI catastrophic risk coming out of the AI community, it appears the AI industry's safety to capability investment ratio is far less than in other industries*. Just treating AI as a 'normal' new-technology safety problem would apparently be a big win *not peer review quality analysis, just had gpt-5.6 look at the sources listed below and do some estimates