I’ve said it once and will say it again; compute superiority is not actual superiority. AGI is not something to be concerned about, despite how much the drama queens of today fear-monger about it (which is just their marketing, anyways). AGI comes about because we’ve captured some notable portion of the process of human thought in a mathematical equation, then trained and fueled that mathematical equation with so much compute we manufacture something akin to a semi-human under the influence of the most powerful amphetamines imaginable. And even if that already sounds concerning to you, LLMs are so far from representing any actually significant portion of human epistemology, and operate with such kludgy bloat, that the molehill is more like an anthill. LLMs basically model the human faculty of linguistically-guided intuition, able to “outperform” us only on purely computational grounds. Think of it this way: if I played against Stockfish (the world’s best chess AI), given enough time to calculate all the needed branches of the game tree, I could reach a similar level of performance, and maybe beat it. The framework Stockfish uses to calculate chess moves is perfectly accessible to me — the only advantage it has is the higher efficiency and energy of the computer in which it’s stored. Given our advances in neural programming, and drugs, I’m not concerned about humanity’s chances here — our biological compute capacity is only increasing. And even then, we seem to be good at getting compute-superior entities under our control without matching their compute capacity directly (nature is the big example here). There is one case where I am concerned, though: architectural superiority. This is not a compute advantage, but a foundational structural advantage. This is unique and unachievable/incomprehensible ability not in the amount of computation, but in the way computation itself occurs. Think of epistemological architecture as the mark of distinction between species — I’d argue humans have architectural superiority to worms; humans can think in ways fundamentally different and fundamentally superior to worms, and the advantage is crushing. If somehow an AI achieves this particular kind of superiority, meaning it is no longer a model of human intelligence, and perhaps could be called a different species, then our safety could be under question. Yes, we have tamed the mammoth or the ape, but that is humanity overcoming computational superiority, not architectural superiority. An AI with the latter advantage would be unimaginably, able to think in ways procedurally and topologically superior to ours. A computationally superior AI could spend a lot of energy and time describing a cube; an architecturally superior AI could fully and deeply comprehend the fourth dimension. My only wondering is if we should be worried about architectural superiority at all: can we even bring something like it about? Could we even realize it existed? My intuition is that recursion provides the most viable path to architectural superiority. But even if we try to create it by recursive processes, is the initiation and base terms of the recursion not something tethered to us and thus indicative of architectural equality/inferiority anyways? Or is there a level of incomputable chaos we could achieve in a recursive engine that may yield an emergent power akin to what I describe? I do not know.