most people here have thought long and hard about ai, every conversation i have — even with those i vehemently disagree — feels like it makes me smarter.

truly an excellent conference. I considered changing travel plans so I could stay an extra day.

Main application window for The Curve ends this Friday, 8/22! We’re super excited to get this excellent and ~eclectic group of people in the same room, discussing toughest questions about the future of AI. Application + more about who will be there below ⬇️

The Curve 2025 is on! We're bringing together people from a broad range of fields & with disparate perspectives to discuss big questions about AI's future (timelines, natsec, growth, applications, & more) October 3-5 | Berkeley, CA Apply by July 4th for early bird pricing

The origin story of “AI as Normal Technology”, and lessons learned Many people have asked how the “AI as Normal Technology” paper came to be. This paper has been an (ongoing) journey for me and @sayashk in developing not just the substance of our arguments but also learning how to engage in productive debate on polarizing topics. So let me tell the story of the paper and the lessons we learned from writing and publishing it. My initial attitude About 2-3 years ago, my writing on social media (and occasionally on the AI Snake Oil newsletter with Sayash) was dismissive of AGI and existential risk. I saw the whole discourse as being driven by Lesswrong / EA / Silicon Valley bubbles, self-serving, and harmful because it distracted from more immediate concerns. A Princeton colleague was peeved by my attitude and wrote to me about it. They pointed out that social media mud wrestling is not a good strategy for scholars and pushed me to defend my views instead of dismissing contrary views. Decision to engage I took this advice to heart, did some reflecting, and concluded that (1) Superintelligence-related ideas are intellectually serious, sincerely held, and deserve to be engaged with rather than dismissed or ridiculed. (2) Social media doesn’t incentivize or enable serious engagement, so I would have to find other venues. (3) It’s true that the AI safety community is insular in many ways, but so is the AI ethics community that had shaped my own thinking; just because someone’s in a bubble or has different epistemic practices doesn’t mean they’re wrong. (4) Besides being the right thing to do, engaging is also the strategic thing to do. Some people were against engaging with the safetyist perspective in the hope of denying it a “platform”. This didn’t make sense because x-risk fears were already mainstream and had a far bigger platform than what a few academics could provide or take away. Initial responses As soon as we started writing up our views and engaging with the superintelligence discourse, we found that our disagreement was more profound than we’d realized — not just at the level of specific predictions or timelines but at the level of worldviews. As our thinking solidified, we did a rewrite, and submitted it to the “position paper” track of the International Conference on Machine Learning (ICML) in early 2024. This version of the paper was narrowly focused on x-risk and policy. We were surprised by the peer reviews. The reviews and the back-and-forth discussion that followed were collectively longer than the paper itself. The reviewers were extremely polarized. The most negative reviewer seemed angry about the paper and argued at great length that it should not be published. The most positive reviewer pointed out that the goal of the position paper track was to generate discussion, and that the discussion that the paper had already generated during the review process showed that it met this bar. But the negative reviewer got their way, as they usually do. In retrospect this was definitely the better outcome for us, because the paper at this stage was far narrower than what it would eventually evolve to be, which is an overarching framework for thinking about the future of AI. At that time we felt we had to get the paper out right away because the x-risk debate was so urgent, but in retrospect we were wrong. It’s always tricky to strike the right balance between speed and completeness. Gaining strength The fact that we and the negative reviewer seemed to totally talk past each other convinced us that there hadn’t been much productive debate between different worldviews on AI safety, and that this was sorely needed. Debates within the AI safety community happen between those with a high p(doom) and those with a low p(doom). But in our view, these two groups are actually very similar to each other — to even consider p(doom) a useful construct is to accept many assumptions about AI risk that we disagree with. The community was also used to criticism from outsiders, but this tended to be intellectually lazy stuff like “but what about the benefits of AI?” or ad-hominem attacks. Real debate was lacking. At the same time, we realized that the paper’s negative reviewer was correct to point out that many of our claims lacked sufficient justification (which is something we’d done to keep the length of the paper tractable). So we spun out separate essays based on a couple of points we’d made briefly in the paper: “AI safety is not a model property” and “AI existential risk probabilities are too unreliable to inform policy”. These essays had a huge impact in the policy debate, which energized us. An experiment in debate-oriented teaching Last Spring I co-taught a course with @msalganik on Limits to Prediction. The class had a unique format. Matt and I agreed on many things and disagreed on many things. We relegated the areas of agreement to the readings and spent class time hashing out our areas of disagreement, inviting the students to participate. The success of this format convinced me that this kind of debate — where people are trying to learn from each other instead of performatively score points — is one of the best ways for everyone to improve their understanding of contentious topics. Anyway, one of the topics discussed in the class was AI x-risk, and we are grateful to Matt and the students in the course for their feedback on a draft of the paper. We continued getting more feedback on our draft, explicitly reaching out to people who disagreed with our views. This really helped because it helped us identify cruxes of disagreement, forcing us to dig deeper and deeper in order to excavate our own and others’ assumptions that hadn’t been articulated. The importance of nontraditional publishing venues Last fall we were invited by @sethlazar and Columbia’s Knight First Amendment Institute to workshop our paper at the Artificial Intelligence & Democratic Freedoms symposium, and publish it on the Knight Institute website. I’d previously spent a sabbatical at the institute, so I knew this would be a great way to get feedback. It also made sense as a publishing outlet. AI as Normal Technology is a nontraditional paper, and I’ve had a lot of painful experience submitting nontraditional papers to traditional publishing venues and finding that they are totally ill-equipped to handle them. Meanwhile, many nontraditional venues such the Knight Institute have strong standards (including, in this case, peer review), as well as rapid turnaround and greater editorial flexibility with regard to length, format, etc. For those who care more about impact than building one’s CV, the choice tends to be pretty clear. Before submitting the paper we did another rewrite to reflect its broader scope beyond existential risk. One thing I’ve learned over the years is that writing is not like the Ship of Theseus. Sometimes the core vision behind a paper undergoes a shift that’s big enough that you can’t get the writing to where you want by incrementally editing the paper; you have to start from scratch and borrow text from the old version as necessary. We published the essay two months ago. We are grateful to Seth Lazar, Katy Glenn Bass, @ghadfield, our anonymous peer reviewer, and many other amazing scholars who supported the paper and gave us feedback throughout the process. Maybe pluralism can work We’ve been thrilled by how seriously people are taking it. Of course, not everyone agrees with the message of the paper, but we’re glad that even those who disagree seem to think it’s a useful contribution to the discourse. We’ve been very surprised by how many people took the time to send us thoughtful feedback. We’ve had many productive conversations to identify points of policy agreement despite differences in beliefs and values, identify the right empirical questions that might tell us how best to prepare for the future of AI, and so on. (Most of it has been happening through channels other than social media.) And we’ve started many interesting collaborations with people who do share the Normal Technology view to figure out the implications for AI’s impact on various sectors and spheres of life. Overall, it has made me more optimistic about the future of AI policy. And at the risk of sounding extremely naive, maybe if individual policy communities get better at good-faith discussion across deep differences and divides, my hope is that we will eventually start to get better at pluralism at the level of society as a whole. I’ve enjoyed following the cautiously optimistic writing on this topic from sources such as @jonathanstray's Better Conflict Bulletin. Lessons learned * Growing from critical feedback — it’s important to be able to ignore haters on the internet while also recognizing valuable feedback from critics. * We’re all in our own bubbles. Good-faith engagement with other communities who have different worldviews is hard and requires a new set of skills, but can be worth it. * There are many tradeoffs to navigate such as speed vs thoroughness, revising vs rewriting, and publishing one big essay vs piecemeal. The answers are rarely clear but it’s important to make deliberate choices instead of just taking the path of least resistance. * The medium matters — in our case, a nontraditional publishing venue over traditional ones, and more intimate discussion settings over social media.