Okay, here it is, the Bad AI Consciousness Takes Bingo blog post, with rebuttals for each and every square and some additional honourable mentions! (I'd like to thank @united for making this post possible, and for the extra pretzels) open.substack.com/pub/polytr…

This really is the key point re capabilities a lot of doomers seem not to engage with. You do not in fact have to do naive statistical extrapolation of low-n, low-external validity benchmarks that are in the dictionary under "Goodhart's law." There is other empirical evidence about capability trajectories, from better benchmarks to microeconomic productivity studies to breakdowns like the below of compute allocation that you can use to make forecasts. I find it very telling that doomers seem not to have updated at all on the facts that: 1) almost the entire last year-and-half of progress stems from math and coding RL, which are dependent on abundant human data AND synthetically generateable data AND formally precise reward signals; 2) the vast majority of tasks that make up white collar jobs do not satisfy ANY of these three criteria. This is not the end of the story of course, as data foundries' whole business model is to try to falsify 2). BUT that is insanely hard! If you've thought for 2 seconds about intellectual labor, you know that it is almost always infeasible to decompose it into bundles of necessary and sufficient success conditions, much less ones that can be formalized to a sufficient degree for RL. You also know that the nature of that labor is in a constant state of change, especially during a general purpose technology boom! Admittedly this is a scorching hot take, but I think these factors alone are sufficient to rule out futures containing uncontroversial ASI or AGI, so long as these are defined as (respectively) parity with or supersession of ~all human intellectual faculties. And that's before modeling the bad politics and incentives around pushing model capabilities into arenas the vast majority of people would rather have them not go.

What if the jagged frontier is mainly math + code (which you can push arbitrarily far with RLVR), and everything else starts to plateau because it is still bottlenecked by human generated data? Model performance in non-verifiable areas has kept improving steadily, albeit much slower than for math and code. But is that steady improvement a side effect of a higher G (itself driven by RLVR), or only a function of the amount of new human data getting injected into training (which is still continually happening on a massive scale)? A lot of things depend on the answer to this question

From a Wittgensteinian perspective the AI consciousness debate is a philosophical nadir for culture. Virtually everything said about AI consciousness represents what Wittgenstein called the metaphysical use of language: words acting like an idling engine, doing no real work. 1/11