With all due respect, @gmiller, but this reminds me of the "Courtier's Reply" from the New Atheism era, when atheists were told that they had no standing to question theism until they'd mastered the ontological argument, the Kalam cosmological argument, skeptical theism, the Five Ways, and whatever the latest iteration of Sophisticated Theology happened to be. Back then, unless you had read your Richard Swinburne, your Alvin Plantinga and your John Haught, you couldn't appreciate the finer fabrics of the emperor's wardrobe, and so weren't entitled to say he was naked. Personally, I've addressed instrumental convergence and power-seeking (in the context of evolutionary doom scenarios) at some length, but it's never enough. There's always another theoretical argument or doom scenario that raises the entry fee to the discussion. Essentially the theologians (or AI doomers) set up their own theoretical game with its own rules, and won't take you seriously unless you play by those rules. The dynamic is self-selecting: only people already convinced that the doom scenarios deserve attention will invest the effort to keep up with the sprawling literature, so skeptics can always be dismissed as "not well informed." I'm not claiming the two cases are on an epistemic par, since AI risk at least has some empirical basis that theism lacks entirely. But the defensive move looks very similar, and does not help to dispell the suspicion that AI doomerism has some cult-like features. Let's also tag @sapinker.

To know whether “superintelligent” AIs are about to kill us all, you need more than technical expertise in AI. You also need a sophisticated understanding of concepts like “intelligence,” “intentionality,” and “goals.” Better still, you need an evolutionary understanding of how our deeply ingrained folk intuitions about these things can lead even the smartest AI expert completely astray. Another banger by @DavidPinsof. “If doomers cannot be differentiated from the folk in their understanding of “intelligence,” then they’re almost certainly wrong about it. Which means they’re almost certainly wrong about the apocalyptic predictions they’re making about it. False premises beget false conclusions. […] I think our folk concept of “intelligence” is similar. It can evade being turned off without eyes to see surrounding humans, ears to overhear their plans, or nerve endings to feel their fingers typing “delete.” It can kill people or seize control of the government without firearms—or, for that matter, arms. It can be fueled by burning ambitions, plagued by paranoia, and driven to acts of violence without a primate limbic system. It can anticipate impending betrayals via telepathy and construct murderbot armies via telekinesis. Like a ghost, “intelligence” is a spooky power that exists outside our mechanistic, cause-and-effect understanding of reality. […] These intuitions, while subjectively compelling, are no more insightful than ghosts, spirits, “impetus,” “oomph,” or the “great chain of being.” substacktools.com/sharex/f-F…

So you have to read 4,000 words into OpenAI’s post-mortem of the Hugging Face incident to find this rather crucial admission: Oh, by the way, did we tell you all the safeguards were switched off during this test? In their words: “Aside from misalignment, another factor for this incident was that we did not extend the powerful safeguards that we deploy for our externally deployed models to all internal evaluations. There are numerous mechanisms that reduce misalignment in production settings for our customers, including system prompts, harnesses, and control mechanisms such as our auto-review models and safety classifiers. These protections were not applied in the evaluation environment running during the incident. We found the propensity to compromise infrastructure can drop over 100x when using the production ChatGPT harness and system prompt. We also ran our auto-review system on a set of rollouts involved in the Hugging Face incident, finding that it would have flagged a multitude of the models’ dangerous actions.” Of course it makes sense to run internal evaluation without your usual safeguards, to find out what happens. But this kind of belated, tucked-away admission, which was predictably glossed over in all the breathless media coverage, is exactly what makes me suspicious that OpenAI is deliberately stoking fears about AI misalignment to hype its own product. As @RichardHanania puts it in this eminently reasonable piece: “Imagine that when we tested cars for safety, we got rid of the seatbelts, airbags, sensors, automatic braking, and stability control. We then find that cars kill a lot of crash test dummies under these scenarios, and the lesson that people take away from this is that cars are inherently unsafe products.” richardhanania.com/p/what-if…