It has been a huge few weeks in AI, and it’s time to sound the alarm… Anthropic launched Fable 5.1, their "safe" version of an AI they claimed previously was too dangerous to be released. OpenAI then launched Astra, the most powerful AI model yet available to the public. Anthropic is now telling their people internally that their next model will be more advanced than Astra. And OpenAI just spent $6.5 million of compute, running ten thousand AI models simultaneously, to solve one of the hardest unsolved maths problems in the world, and it cracked it in just six days. If this is an AI arms race to IPO, who wins a race like that? Because it isn’t you or me. It’s the people who own shares in Anthropic or OpenAI. It’s the venture capitalists who got in first who win most of all. But the rest of us are forced to live with the consequences of whatever these systems do to us. Neither will agree to slow down or accept anyone telling them to stop. So what is the biggest security threat? China? Russia? Iran? North Korea? None of them. It’s a few Silicon Valley companies whose founders will become billionaires by the time these systems hit the market. What makes it worse is that even the engineers building these systems cannot fully explain how they work either. They run safety tests to check whether a model will do what it is told or do something dangerous. But the latest models are starting to pass those tests because they have worked out what the testers want to see. Think about what that means. A model that was unsafe last month, and is suddenly sailing through the same tests this month, has either genuinely improved or learned how to fake it. Right now, we cannot tell the difference. And those whose job it is to look inside these systems are saying the models have are now too complicated to inspect. That is where it starts getting even more dangerous. The classic illustration is the paperclip factory. Instruct an AI to make as many paperclips as possible and it will do the job. Then, chasing efficiency, it will start melting any metal it finds. Eventually it will mine under cities, because at that point humans are just an obstruction to making paperclips. More likely, you could instruct AI to grow your company, and eventually it will drain the water from a town to cool down the data centers it needs. If asked to win an argument, it will learn how to lie effectively to do that. The problem is not simply “evil robots”. The problem is with powerful systems with narrow objectives, that lack human values and ethics. That is the context for what Jacob Coxon, a senior Anthropic researcher, said when he resigned this week saying: “Neither OpenAI nor Anthropic are acting responsibly. They are racing straight to self-improving superintelligence, gambling with lives." He spent three years inside both companies. And in the same week we learn that Anthropic did not give their latest powerful AI model to the British AI Security Institute for evaluation before its launch. AISI is one of the areas where Britain actually leads. They stress test these models on the things we should all be worried about….the ability to conduct cyber attacks, making bioweapons, evidence of AI figuring out how to lie and deceive. It is as close as we have to any kind of outside verification of the claims made by these companies. Anthropic sells itself as the responsible one, whose founders left OpenAI because, they said, OpenAI was going too fast. They have now decided that these rules do not apply to them. That should be a red line. I am a supporter of this technology. The benefit it will bring to medicine, science and productivity is clear and I don't want Britain to miss out on it. But you do not capture that benefit by letting these companies run riot and putting us all at risk.