Philosophy of mind discourse on Twitter is the specific place I consistently see otherwise very smart people say the equivalent of "Well moron rocks could never orbit the Earth because their telos is to be close to the ground, everyone with half a brain knows that."

Pinker's recent media run arguing against the idea that catastrophic AI scenarios are worth seriously considering is probably the fastest any public thinker has ever fallen in my esteem. It's not just that I think he's wrong; it's that he plainly has not bothered to even *read* opposing arguments or even background material on the field. He does not know the terms of debate. This clip provides an example. "Superintelligence" is, famously, a term popularized by Nick Bostrom over the last couple of decades to refer to an AI system that dominates the best human capabilities in all relevant domains. All concepts are in a sense "made up," but this concept seems quite useful for conceptualizing possible futures and human-AI interactions as progress continues. If one wants to argue against that concept being possible or useful, they can, but they should be aware of the sense in which most people (doomer, e/acc or otherwise) use the term. Pinker isn't. He immediately jumps to the notion that "superintelligence" must mean total omniscience and omnipotence, to the point of knowing the location of every particle in the universe. He scoffs at this and rejects out of hand the notion of "superintelligence," going on to say that the labs are therefore peddling hype, and the doomers mystical nonsense. The threshold he sets is obviously so far above relevance to human scale that it is essentially meaningless, and I can't help but think that anyone arguing charitably would, you know, notice that and look deeper into how the term is used. Pinker doesn't, because he seems to have some very strong pre-existing bias against AI extinction risk as a possibility, and accordingly refuses to even investigate alternative positions. There are many other examples of this. In his recent open letter, he takes for granted that superintelligent systems will not have any kind of internal, intrinsic motivation, ignoring that increasing persistence and autonomy are key aims of those developing AI, because, just as predicted, they are incredibly useful traits for accomplishing tasks that are valuable to humans. It's hard to read transcripts from the HuggingFace incident (or even just the running commentary consumer AI provides while working), and not see that we are deliberately instilling motivations into these models. Similarly, he rejects the notion that we need worry seriously about AI systems having autonomous control over important infrastructure or military technology, because for humans to hand over such power would be "stupid," and therefore will never be done. He accordingly feels he can handwave away the considerable thought and writing given to gradual disempowerment scenarios. If he did read them, he'd see the insight that, as model capabilities surpass humans in various domains, competitive pressures will inevitably force actors to hand over increasing control to AI systems whether they want to or not. The "man in the loop" may be a luxury you can't afford if a competing power achieves locally better results by eliminating it. Today he bandies around the word "cult" to dismiss those concerned about catastrophic AI risks, while he refuses at every turn to engage with them. Comments are kept turned off. Debates are declined. He does not even commit to a sustained written exchange. This is a man I once truly respected, whom I was honored to shake hands with a few years ago. My best guess is that he has wedded himself to his heuristic that human society steadily improves over time, and is responding defensively to avoid confronting the notion that this guideline may indeed break down in the near future. But nothing excuses his atrocious conduct on this enormously important issue, and his stunning failure to live up to the enlightened standards he has long trumpeted.

Dario Amodei is at the desk to assure that the future of humanity is safe from AI

In his letter to Scott Alexander, Steven Pinker lists four objections to AI extinction scenarios. Here are his points, followed by my responses. 1. "They had an underdeveloped conception of intelligence which treated it as a quantity of power which may be extrapolated from animals to dull humans to smart humans to AI to Artificial Superintelligence, the latter consisting of perfect omniscience. (This was the focus of my later exchange with Scott Aaronson.) I argued that any intelligent system is a mechanism which is good at solving some problems but not others, and which is inherently limited by knowledge about the world attainable only by observation and experimentation." No one disputes the need for knowledge. AI systems are already absorbing vast amounts of it from books and other sources. We are now connecting them to biological laboratories so they can learn through experiments. These are precisely the ways of acquiring knowledge that Pinker says intelligence requires. 2. "They conflated intelligence with motivation, particularly self-preservation and dominance. I argued that these motives happened to come bundled with intelligence in Homo sapiens because we are products of natural selection, but they are not inherent to intelligent systems that are engineered." Pinker is missing instrumental convergence, the most important concept in AI risk theory. Whether an AI wants to study mathematics, explore the stars, or help pigs, being shut down would prevent it from pursuing those goals. More resources and power would help it achieve them. It therefore has reasons to resist shutdown and acquire power, even if neither is something it values for its own sake. These are instrumental goals: means of achieving other ends. Evolution favored similar drives in humans because they helped our ancestors pass on their genes. An AI need not share our evolutionary history to face the same incentives. 3. "They assumed that an artificially intelligent system would monomaniacally pursue a single goal, heedless of side effects. I argued that trading off multiple competing goals is the essence of intelligence, so no genuine AI would wreak the ridiculous havoc imagined in the doomer scenarios." The AI risk argument does not require a system to pursue a single goal. We use paper clip maximization because it is an easy example to explain. An AI could balance many competing goals and still take our resources to pursue them. The ability to make intelligent tradeoffs among its goals does not mean those tradeoffs will protect us. 4. "They assumed that human engineers would grant these systems unlimited and irreversible control over the earth’s physical infrastructure, amounting to omnipotence over every atom on the planet." We do not assume that engineers will voluntarily give AI unlimited, irreversible control over Earth’s infrastructure. Earlier discussions considered whether even an oracle, confined to answering questions and isolated from the internet, could be safely contained. The amount of access we are already giving AI is astonishing. But the danger does not depend on our handing over control voluntarily. A system smarter than us could acquire power we never intended to grant. Lions did not give us permission to take their territory. If AI becomes smarter and more powerful than humanity, it could take over.

If you thought mass immigration by humans was bad, wait until you understand mass immigration by AIs... MAGA conservatives have opposed mass immigration & open borders for three main reasons: 1) mass immigration takes jobs from Americans 2) immigrants who don't respect our norms and laws impose serious security risks on Americans, 3) immigrants with very different values & beliefs, who don't assimilate, will change American culture in bad ways. All three reasons apply even more strongly to any mass influx of agentic AIs. @KatjaGrace, below, is right about this. 1) agentic AIs will take our jobs 2) agentic AIs that aren't aligned to our norms and laws impose serious security risks on us 3) agentic AIs with very different values & beliefs will change our culture in bad ways. I made similar in my talk at @NatConTalk last year, when I said "Many conservatives in the current Trump administration, quite rightly, want stronger geographical borders against alien people. But they seem oblivious about protecting our digital borders against invasion by alien intelligences. Indeed, they seem giddy with delight about AI companies growing ASIs [Artificial Superintelligences] inside our data centers – without understanding that a few ASIs can easily become hundreds of ASIs, then millions, then billions. If you worry about immigrants out-breeding our native populations, wait until you see how quickly ASIs can self-replicate. These ASIs won’t be American in any meaningful sense. They won’t be human. They won’t assimilate. They won’t have marriages or families. They won’t be Christian or Jewish. They won’t be national conservatives. But they will take our jobs." Many of my followers supported Trump's slogan in 2016: "Build the Wall!' This may be time for 'Build the Wall 2.0': stop agentic AIs from flooding our country with economic competitors, dangerous rogue agents, and AIs that have alien values & beliefs. tldr: If you oppose mass immigration by millions of humans who can't or won't assimilate to American values, you cannot support mass immigration by billions of AIs that can't and won't assimilate.

More than anything since Hugging Face, this podcast raised my p(doom). A very smart guy, with a very big financial interest in the intelligence explosion, spouting very, very much nonsense to downplay the risks. 1/5

This is insane!