This illustrates the point I’ve been making about the limits of centralized approaches to SI safety in cyber. OpenCode (and earlier Hugging Face) hit the same wall mid-attack: frontier models refused to analyze the very exploits needed for defense because the payloads tripped safety filters. The guardrails backfired. Meanwhile, the attackers found a way around restrictions to access advanced capabilities. This is not surprising. Chinese models are only about 6–9 months behind the American frontier in cyber. Obviously the U.S. has no jurisdiction over them. The only durable response is decentralized: make powerful models available so defenders can build and run their own cyber defenses. The only solution to SI-powered cyberattacks is SI-powered cyber defense.

Are Small Models Really a Threat? The Anthropic Fear Factory just released a new report warning that small models may be only months away from self-replicating across cheap IoT devices like off-brand Android TV boxes and smart ovens. They say such an AI worm would be hard to shut down since there’s no central kill switch. We’ve gone from “The Terminator is coming” to “your toaster might kill you.” But the experiments don’t support the dramatic headlines. They ran a 26B model across 15 $55 TV boxes at 0.39 tokens per second. The end-to-end test used emulated devices with deliberately open ADB ports, so the “hacking” step was trivial. The capable model often ran on a separate server. Agents were harnessed and told to propagate. This is not proof of autonomous rogue AI worms in the wild. Nor is the threat even novel. Worms that do this without AI already exist. Mirai in 2016 scanned for default credentials and built massive DDoS botnets. Kimwolf has infected over two million devices, mostly cheap Android TV boxes, by hitting open ADB ports. Small models don’t currently add much that matters on these targets. The bottleneck has never been figuring out the device. It’s the huge number of devices that ship with open ports, default passwords, and no patching. Traditional worms already exploit that at scale. A practical attacker would use the best models available offline to design a better traditional worm, then ship the lean binary. You get the intelligence benefit at design time without dragging a slow, heavy model along with every infection. Embedding a weak model in the payload is the worse option for anyone who actually wants the largest, fastest botnet. The timing of Anthropic’s latest scare campaign is hard to ignore. The day before, Brian Armstrong posted that Coinbase fine-tuned an open-weight small model (Qwen-3.5-9B) on proprietary fraud data. It beat frontier models, including Anthropic’s own Opus 4.5, while being faster and much cheaper. In testing it cut fraudulent transactions by about 30%. This is the emerging pattern. For narrow enterprise workloads you don’t need a general superintelligence. You need a small specialist model trained on your own data. Cheaper to run, especially as compute gets tighter, and often better on the metric that matters. That is a direct threat to the frontier-company model of selling access to ever-larger general models. Safety advocates should be welcoming this development. If the real concern is existential risk from ever-larger frontier systems — the ones that could theoretically do sustained autonomous R&D or become hard to control — then a shift toward smaller specialized models is the safer outcome. Small models specialized on fraud detection or support tickets are not pushing the frontier. They are not in the category of systems that could become a new species. The theoretical loss-of-control risk is created by models getting bigger and more generally capable. Safety advocates focused on those risks should welcome companies having a cheaper alternative to constantly scaling general models. Instead we get dramatic warnings about small models at the exact moment they are starting to take real enterprise share from closed-model companies like Anthropic.

Anthropic says it has reduced the data harvested from Claude in distillation attacks by around 90–99%. I congratulate them on this achievement. It’s consistent with the point I made a few months ago (All-In Ep. 282): Anthropic, not the government, was in the best position to stop distillation of its own models. Once the horse is out of the barn, there’s limited value in after-the-fact government action. At the time there was a major push in Washington to restrict open models on IP grounds. The threat looked serious enough that 50 companies — including Nvidia, Microsoft, Meta, IBM, Palantir, and Dell — published an open letter in defense of open-weight models. Jensen Huang promoted it in his first-ever X post. Nearly 200 startups sent a similar letter. I argued that restricting open models risked removing the incentive for closed-model companies to solve the problem themselves — and could reward negligence by stifling competition while they continued collecting revenue from distillation. The administration did not impose those restrictions. Now we see the result. Anthropic invested in defenses and they worked. Not every technical challenge is a reason to run to Washington for new regulation.

Important piece from Microsoft AI CEO @mustafasuleyman. Training models to think of themselves as human (as Anthropic does Claude) with concepts of an inner self, psychological wellbeing, rights and presumably grievances, is a potential existential risk. project-syndicate.org/onpoin…