The belief that AI would eventually herald the end of humanity is not a new one. It has not arisen in response to the release of the LLMs of ChatGPT and Claude. It has not emerged as a response to recent technological developments. The story starts in the 90s, with @allTheYud. A precocious youngster with no formal education, he joined an obscure internet mailing list [created by @perrymetzger] devoted to futuristic ideas ... There he began thinking about “superintelligence.” At first Yudkowsky wanted to help create this so-called superintelligence... & ... helped establish an institute devoted to the project. It’s unclear when his optimism turned to fear, but at some point he came to believe that a superintelligent machine could escape human control and, unless it shared our goals, destroy all of humanity. Yudkowsky developed these ideas online in a series of essays and attracted a community of like-minded fellow travellers ... His prolific writing became influential among people working in Silicon Valley, many of whom work in today’s AI companies. Among the Rationalists, the imminent arrival of superintelligence – and the end of humanity that follows – is axiomatic. This conviction is shared by many people who live together in the San Francisco Bay Area practising highly unconventional lifestyles. It is not at all surprising that a community in San Francisco would share kooky or even apocalyptic beliefs – the region has been the home of eccentric subcultures for generations. But what is surprising is how far that world view has spread. In Australia, Yudkowsky’s recent book, If Anyone Builds It, Everyone Dies: The Case Against Superintelligent AI, co-written with Nate Soares, has been widely discussed in media circles. Journalist @hughriminton and ABC chairman Kim Williams both have described it as “compulsory reading”. Not everyone accepts this premise. I emailed @sapinker, to ask what he thought about superintelligence. “‘Superintelligence’, with its comic-book prefix, is more a fantasy than a coherent concept,” he wrote. “People use it as a synonym for ‘omniscience’, imagining a magical wizard that can solve all problems with pure computation. Or they imagine that the IQ scale that differentiates humans within their natural range of variation can be extrapolated indefinitely upwards. But real problem-solving requires massive amounts of knowledge about the messy, chaotic, world which divulges its hidden workings at its own pace, only through laborious experimentation. And human intelligence is not some elixir that you simply have less or more of – it’s a gadget that evolved to solve some problems with ease and others laboriously or not at all. “AI is a different kind of gadget with its own profile of strengths and weaknesses, not an enchanted brew that can grant any wish.” @GaryMarcus, a cognitive psychologist and machine learning entrepreneur, likewise believes the risk of AI leading to human extinction is virtually zero. “Humans are too geographically spread out, too genetically diverse and too resourceful to simply fall apart altogether,” he writes. “The idea that AI will kill us all in five years is preposterous.” Marcus does not argue that AI is harmless. He worries about AI being used to develop biological weapons, launch cyber attacks, spread disinformation and enable authoritarian governments – all risks that are catastrophic, if not existential. But there is an important difference between risks we can observe and risks that occur because of human negligence or malevolence, and a chain of events that exists mainly in our imagination. Part of the disagreement stems from the language we use when we discuss AI. @MelMitchell1, a professor at the Santa Fe Institute and author of Artificial Intelligence: A Guide for Thinking Humans, has criticised our habit of describing machines as if they were people. We often say an AI “thinks”, “believes”, “lies”, “schemes” or “wants” something. These words are a convenient shorthand but they also can create the impression that software has become an independent creature with intentions of its own. Mitchell makes this point when discussing the recent Hugging Face cyber-security incident, in which autonomous OpenAI agents escaped their “sandbox” – a computer environment isolated from the internet – and hacked a real-world server. “First, OpenAI did not have proper security measures in place,” she writes. “They turned off safeguards built into the models, instructed the models to find and exploit software vulnerabilities, and let the models run autonomously for weeks without sufficient human oversight.” Rather than showing autonomous AI going rogue, the incident demonstrates what can happen when humans give powerful AI systems dangerous instructions without adequate safeguards. AI is, of course, advancing rapidly. Machines can write computer code, translate languages, diagnose diseases and solve some of the hardest problems in mathematics. But to get from the AI we have today to the extinction of humanity requires several further links in a chain, none of which are guaranteed. Philosopher @mboudry, writing in @Quillette, offers a useful way to think about this. Humans (and other animal species) evolved to have a competitive drive, sometimes manifesting in selfishness and aggression, across a time span of millions of years. Our ancestors survived because they fought hard to secure food and mates. Out in the wild, these selection pressures led to the evolution of traits that enabled animals to hunt and capture their prey. But artificial intelligence does not exist in the wild. It was created by us and exists in the equivalent of a petting zoo. And just as we have been able to domesticate wheat for our food and breed dogs to be our companions, we are able to select the conditions under which AI develops. We are not selecting AI models on the basis of their ability to hunt prey in the physical world. We select them on the basis of how helpful they are to us. “We have been selecting chess computers for cognitive capacity for decades,” writes Boudry. “Their capabilities now far outstrip even the most gifted human grandmasters, yet they have not become harder to control.” Boudry accepts that an AI could slip out of human hands one day, through accident or malice. Even then, he argues, the likely result is not extinction but something like the long battle between computer viruses and antivirus software: costly and ongoing but not the end of the world. The problem with apocalyptic fears is that when they become mainstream, they can be hard to wind back – even in the face of contradictory evidence. Across the past 100 years, apocalyptic anxiety has leapfrogged from nuclear annihilation, overpopulation, environmental collapse, to rogue AI. (Some of the dangers behind these warnings were very real, of course, and some of these risks remain.) But we also have to ask what happens when this anxiety becomes locked into public policy. The ban on nuclear energy in Australia is the most obvious example of the damage this technophobia can do. Australia has 28 per cent of the world’s known uranium resources and has exported uranium for decades. Australian engineer Bobby Gallagher has invented a nuclear reactor that can be deployed on the back of a truck, a technology that has been hailed by Trump. Yet this form of clean energy remains prohibited in our country under federal law. Public anxiety surrounding nuclear weapons, radioactive fallout, accidents and waste means that while Australians can mine uranium, put it on ships and sell it to countries that use nuclear power, we cannot build commercial reactors for ourselves, using the ingenuity of our own people. The situation is a disaster. And in an age of superpower rivalry, technophobia does not remain purely a domestic matter. During the Cold War, the Soviets promoted a fear of a “nuclear apocalypse” in the West, and provided propaganda and funding for peace groups and antinuclear activists. This does not mean that the millions of people who opposed nuclear weapons were plotting against the West. Most were ordinary citizens sincerely frightened by the possibility of nuclear conflict. But the fear was useful to our adversaries. To weaken democratic nations, foreign powers do not need to invent anxiety or division – all they need to do is magnify it. In recent days Trump has said the US will not be slowing down the development of AI. In Australia, for the time being, Anthony Albanese also has resisted calls to stop AI development, instead promising national rules designed to capture its economic benefits while managing its risks. Both positions are reassuring. While AI comes with danger, we should be sceptical of any narrative that conveys inevitability around its trajectory. AI systems do not build their own data centres. They do not manufacture their own chips or connect themselves to power grids. Humans decide which systems can access the internet, whether they can control machinery, move money or operate weapons. Humans build them, finance them, deploy them and decide what powers to give them. We also should remember that some of the stories we are told owe more to myth than to science. As Nvidia chief executive Huang has said of the AI doomer narrative: “I appreciate that many of us grew up and enjoyed science fiction, but it’s not helpful. It’s not helpful to people. It’s not helpful to the industry. It’s not helpful to society. It’s not helpful to the governments.” Like every technology that has come before it, humans have agency over how AI is used. Instead of adopting a posture of fatalism, we should decide what kind of AI we want to build, what problems we want it to solve, and treat it as a tool rather than a force of nature. Read my full piece for the @australian here theaustralian.com.au/inquire…