PaulGodsmark retweeted
Paul Christiano's Dwarkesh Superintelligence Interview
Dire. Paul thinks there's a 33% chance the US government will be overthrowable by AI in 12 months.
You obviously think now that Paul is a clown. But you are reading my post, presumably you don't think Diego is a clown.
So let me change that real quick. Paul Christiano is the second person I most trust in the world.
The third is Diego Caleiro.
So, with that out of the way, let's get to business. Paul has a lot of information about what the top AI labs are doing, recently joined the board of OpenAI, used to be the USA gov AI Safety guy, invented RLHF, and I met him 13 years ago and had a few dinners with him, visited some of his house parties, took his first ever class on RL in Berkeley when it was merely a very experimental think he was incipiently thinking about (I think there were like, 8 people watching). I noticed Paul's intelligence yet jaggedness the day I met him in 2012, but as any intelligent system, he became less jagged over time and he is able to speak human now fluently. For over a decade I used to say he is the smartest person I ever met, and I've met Thiel, Musk, Pinker, Dennett, Deacon, Tegmark, Minsky, Yudkowsky, and many others. That's relevant context for you to watch the interview (in comment), because when he says anything outrageous that you think you can easily refute, just remember Diego would bet colossal amounts of money on Paul being right, and your refutation or emotional reaction after thinking about it for 15 seconds being wrong.
Now, what is the plan?
There are many different research agendas that look promising.
We define promising here as having like a 5% chance of working. Not in their current form, just take their direction, then give it to a bunch of very super human AIs, and let them improve that direction. 5% chance.
ARC, the thing he is running, is one of those things he thinks in its mature form could stand a chance of 5% of fixing things.
He thinks there are several other things in that 1 to 10% range.
Again to repeat, that's not solutions we already know. That's things we hope that if a supercomputer gets a hold of them in 7 or 9 months, they progress it to maturity, then use to save the world.
His overall guess is that considering all this, 50 50. coin toss, human disempowerment and it's over, 50 we are so back. Separately, he things another coin toss on stopping before capabilities reach these levels. So 75% of good outcomes overall. 25% of doom, give or take.
Good. Good. My P(Doom) has been lowering this year, most recently due to him joining the OpenAI board, and stood before the interview at 69% conditional on not stopping. Since Paul knows more about what is going on than I do, and his is 50%, I'll lower mine..... 4 of those 19 points separating us. My new P(doom) is 65%, hoooray.
The scary part: The whole interview is probably the most terrifying piece of media ever released from the perpective of a normie, but I've known about this shit since 2001-2004 so I would not call it terrifying, it lowered my P(doom) after all :)
But the scary PART of the interview is hearing from OpenAI board member that the literal plan is to make the AIs of 6 to 9 months from now take over the AI safety domain and save us. We're flying a plane and building it at the same time, we hope we can build a robot who is a better plane engineer and pilot than we are on the way. That is literally the plan at the cusp, of the cusp, of the bleeding edge of human ingenuity.
How did we get here? We tried. People like me and Paul have done basically this for 15 years. there's hundreds of other gifted people who have the gift of foresight and can see the future who did the same things. Thousands if you lower to 10 years, hundreds of thousands if you lower to 5 years. We really, really tried. But we are still apes fighting Ancalagon. Forgive us.
Although we are imminently going to die, according to many people in the field, in 18 months or so, one thing I took from the interview is that disasters more broadly will happen a little sooner than I imagined. I was expecting to be biking on my merry way a few years from now, and then die. But now I am a little bit more worried that there will be civil war, food shortages, systems failure, electricity failure, starvation, thirst, government overthrow, a bit sooner than that. Before I thought the ideal strategy for AI is keeping everything normal until it can act all at once to take over the world. Now I'm more inclined to think it is a reasonable idea to fuck everyone completely before that.
bummer. I was hoping not to have pass through the apocalypse on the way to paradise or oblivion.
So what should people do in light of all this? 33% is not 95%, don't go acting like humans lose control in a year, you'll get fucked two thirds of the time. Don't go on a killing spree or spend all your money in drugs, but also don't save for 10 years from now. Build some resilience for the first times the internet and electric grid goes down, find out about the rivers with drinkable water near where you live, get guns and ammo. Don't fight your neighbours, they may be your survival ticket during the collapse phase.
I expect the collapse phase to be absolutely brutal, but very short. Not WW2 like a prolonged event with atrition and whatnot. I think the most likely scenario is the AIs will destabilize the earth just enough to secure Singleton status, negotiate with our institutions like the USA gov, maybe the UN, a few militaries here and there, people with nukes, maybe the EU, who knows.
Then there will be rapid stability once control is permanently seized by them. Probably everyone will be rich UHI, and we will figure out how to live in the new world, while grieving for those who died in the transition.
If things go well, we could skip most of the apocalyptic period, or even all of it. No shortages, not war, just a gradual industrialization and progressive implementation of paradise, starting probably by curing all diseases affecting people who are 80, then 70, then 60, and so on, then increasing wealth of those who are miserable first, then poor, then lower class, until we are all living in emerald castles on the moon. There may be a period of consistent advantage of USA residents, where Government distributes resources for us first before saving the rest of the earth.
The most dramatic change I expect to see from a quality of life perspective, once robotics is superexponential, and one I'm very curious about, is how many people will choose to move to insanely cheap apartments in unnocupied land. If you had to choose between your current location, or a totally shitty location but a 10 bedroom mansion with Amazon drone delivery, what fraction of people would move to the mansion, what fraction prefer to stay in their city? I have zero fucking clue. And I want to find out
I had housemaids until age 27 and highly recomment getting them once they are nearly free and actually good. As I mentioned a few writings ago, I expect to date a robot at some point, but probably not among the first million people to do it.
Because I now see the apocalypse as a bit more likely, I am starting to think the time to make money is over. not for me in particular, for everyone. At some point there's a transition to "stop saving for retirement" to "stop working altogether" and I think we are crossing it in the next 3 months or so for most people reading this, who have high IQ, savings etc... we could be past it.
The AI 2027 guys are guessing AI will happen in 2027 or 2028, so is Dwarkesh, Paul didn't say his 50% but looks like he is 2028-2029.
I am a bit slower than these guys, for what I would call "vibe reasons". I've been around for 20 years thinking about this, I built good intuitions. I know some things about human intelligence and evolution they don't know. I have my reasons, but I don't have a ready set go essay about why I think things are going to be slower than they do. This text is taking them seriously, and I do, but at least when it comes to timing, this text is their predictions, not mine. Mine are slower.
Good luck to us all.
How are you concretely changing your life in light of all this?
.@paulfchristiano thinks that on the current trajectory (absent any slowing down), there is a 1 in 3 chance that we could have superintelligence powerful enough to overthrow the government within a year. (He articulates a concrete scenario).
He argues that the labs should pause further algorithmic progress until they can secure their weights and research secrets. Otherwise China could simply steal the superintelligence in the middle of the intelligence explosion, and the labs would no longer be able to slow down even if they saw their models going off the rails.
He also explains how AIs will be trained during the intelligence explosion, and why even a system trained entirely inside a data center could still have the capabilities profile such that humans would have very little chance of competing with it or controlling it.
And he discusses the plan for alignment - basically, making a superintelligence that actually does what we ask the way we mean it. Fixing reward hacks and building better graders will help up till the next few model generations, but he thinks that no single current approach has better than even odds of letting us control superintelligence. The hope is that the AIs figure out the solution to this problem during the intelligence explosion.
Paul is head of the Alignment Research Center. Previously, he was head of AI safety at the US AI Safety Institute. Before that, he led alignment at OpenAI, where he invented RLHF. He’s recently joined OpenAI’s board (though here he is speaking as a private citizen).
PaulGodsmark retweeted
It's time to panic about AI.
For years, I listened to the reasonable people. I nodded along when critics called AI “a con,” and when they waved ChatGPT away as “spicy autocomplete.” I laughed in 2024, when the internet was mocking ChatGPT because it couldn’t count the r’s in “strawberry.” When I met young researchers who had devoted their careers to the danger of rogue AI, I thought: what a waste of talent.
But at some point, I started to realize something was off. The machine kept learning things it was never supposed to learn.
OpenAI recently published more than 700 papers with the solutions to hundreds of open problems in mathematics. Hugo Duminil-Copin, winner of the Fields Medal (the highest prize in mathematics), wrote: “The shock is immense. I am paralyzed.”
None of this should have happened. So I went back and looked at what else the skeptics had predicted, and it dawned on me: they had been wrong about... pretty much everything. Next, I looked at the predictions of the crazy people, and realized they’ve been eerily prescient.
Why then, you may wonder, does the world still feel pretty normal? Because new technology usually spreads more slowly than its inventors expect (it took factories decades to reorganize around electricity). But the scary scenario is not about what happens in the wider economy; it’s about what happens inside a handful of companies.
I wrote down what changed my mind, and what we can do about it:
rutgerbregman.substack.com/p…
#BTC I have a totally made up 'Bitcoin power law' set of curved lines, and we already bounced right off of the low corridor line around end of June, and I am now watching with interest whether BTC gives two hoots about my imaginary line. Low corridor line is around $64,800 right now, 'fair value' $129k and resistance (top) around $324k.
All imaginary, but watching with popcorn...
2
138
normalcy bias is your friend
the super exponential won’t just creep on us - the time between sensing it’s arrival and when it is overwhelming could be weeks, days, hours… maybe minutes?
61
In which @AdamFriedland discovers, to his horror, that *his* idea for AI safety is the actual and only plan the companies have for getting vastly super-human intelligence not to kill us
63
couldn't stop thinking about this so i turned it into an interactive chart on my website.
if the AECI is over 188 by jan 1st, we're probably fucked.
a9l.im/#fits
55
i can't talk to friends or family about ASI and what might happen next year without sounding like a chronically online person lost in sci-fi fantasies. it's increasingly frustrating that trying to be rational about what's happening sounds delusional to normies. like i'm the one detached from reality when, from where i'm standing, they're the ones missing what's unfolding right in front of them. the gap in perception is getting so absurd it's making me feel like I am the schizo who needs to touch grass but im just trying to inform them because the media doesnt
69
normalcy bias
[read about what’s happening with AI]
ME: Wow we’re actually legit fucked this is so bad.
[go outside]
ME: Wait those people are all acting like nothing unusual is happening. Maybe I was just in a paranoid bubble? Yeah it must be that. Phew.
[read more about what’s happening with AI]
ME: Wait no we ARE fucked. Shit. It’s all those people outside who are in a delusional bubble.
[go outside]
ME: Wait...
83
there is a reason none of us can possibly conceive of the true meaning of this…
despite our remarkable imagination, there are some things so complex, so dramatic, so disruptive that we simply hit an internal buffer overload
Look at this graph. Notice how it’s an exponential *on a log scale*. I pray and hope there will be strongly diminishing returns before long in this process. But leaders at METR and Anthropic both seem to think there might not be. That’s fucking terrifying.
102
“This report is also some evidence that a software-only singularity is plausible. I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.”
meanwhile so many orgs allow work-AI access to Copilot - not sure that will reach superintelligence within 18 months…
us normies really don’t have a Scooby Doo what the next two years has in store
I've always had substantially longer timelines than my friends at Anthropic — I've been concerned that they're in an echo chamber and believing the hype about their own models too much, and that they're underestimating bottlenecks and unknown unknowns. And there have definitely been specific occasions when they have overstated how much uplift they were getting from their models. Nonetheless, if METR is correct here (and I think they are among the most trustworthy sources on this topic), then my Anthropic friends have been much more right about timelines than me.
This report is also some evidence that a software-only singularity is plausible. I now think that we are more than 50% likely to have full blown superintelligence in the next 18 months, absent a slowdown. I need to call my family.
147
"model-name-hold-my-beer" indeed…
if you thought things are getting weird now
well today is the least weird it’s going to be going forward
Sam Altman on automation of AI R&D, RSI, OpenAI's internal model:
1. "We don't know yet how [automated AI R&D] truly is going to work. We know today how to say: find me a 5% compute efficiency win."
(Note: a 5% compute efficiency gain that the model can find automatically is actually a big deal!)
2. "What I don't think anyone knows how to do yet is to say, go discover a new architecture that is as big of a single leap forward as the transformer was. And I think that's where things are heading."
3. "If you apply the same pipeline that we applied to get our models to be so good at math to AI research, it will probably work [for automated AI R&D]."
4. The bottleneck to getting good AI R&D results is shifting to asking the right questions and using things like swarms correctly, but answers to these questions will become much clearer in the next 3 months:
"One of the interesting things that people talk a lot about in OpenAI... is what are the questions that you're supposed to ask of GPT-7? Probably it's not enough to just say, make me RSI. And also, it's probably not enough to say, buy me a 5% compute efficiency win. I think this is one of the most interesting debates going on right now. And the answer is not just what question to ask it, but it's things like agent swarms. How are we going to use these models in a way that, like, actually maximizes what we can get out of them? I think we're going to learn a lot about that in the next 3 months."
5. OpenAI's extreme focus on model efficiency:
"I bet we could redo Navier-Stokes at 1/100 of the cost a year from when it was done."
6. More on OpenAI's internal model:
"There was one weekend when we got a new model... The name of the channel was "model-name-hold-my-beer". And people were just throwing that at the hardest problems they could. And people just didn't think the model was going to do these things. And we had this collective experience of people's hardest research problems from their academic career, the model was just knocking down one after the other, many different fields, it was incredible to watch... it was many of the problems we published, many other things."
7. The other loop is self-replicating data centers:
"Yeah, people talk a lot about RSI as AI that does AI research. There's very little talk about self-replicating data centers, this idea that you can use AI to control robots and run fabs and build turbines or solar panels and build data centers. But I think we have that within grasp."
1
116
type 1 thinking leads to type 1 model which leads to type 1 comments with distinctive type 1 humour
despite the unfortunate timing, we take disclosures seriously at @typesafeai
at approximately 6:49PM this Friday evening, we became aware of a massive unauthorized leak of ALL customer noul outputs
in effort to maximize transparency, here is the link: gist.github.com/diogo149/810…
1
145
PaulGodsmark retweeted
Mind-boggling - just mind-boggling
Now expected to peak at an extraordinary 4.2C in December
as a total software vibe-coding noob I say with all due respect to highly experienced coders - “this is my world too now; get used to it”
a weird inversion with LLMs is the models improve faster than the tinkerers
when i see people with custom workflows and setups they're all addressing problems that don't exist anymore
the person naively using vanilla codex is more likely to be experiencing state of the art
1
119
PaulGodsmark retweeted
Here's what this year's El Niño looks like next to the three strongest that came before it.
What stands out to you?





























