"agent evals are about to become as important as building the agents themselves" xitter.cf/V1kiiiii/status/211251… This is our thesis as well. I would go further — building an eval *is* what it means to build an agent. In my ICML talk, I argued that as AI capabilities advance, more and more human effort will shift from building to evaluation, in many different domains. normaltech.ai/p/what-will-be…

As LLMs evolved into agents, earlier best practices no longer serve us well. This new research field needs a field guide, so we wrote one. If you're new to agent evals, please take a look. Seasoned researchers and those interested in steering the community will also find many useful observations. sage.cs.princeton.edu/trends…

Sure. If you are looking for true cruxes I doubt there is one. Beliefs tend to be tightly correlated (more than logically entailed), which says something about the discourse. That was part of our motivation for writing the paper.

Here are 14 points about AI safety I made on the Ezra Klein show, and some impressions about the experience. ​​nytimes.com/2026/10/09/opini… 1. Intelligence isn’t power — An A.I. system being powerful is not a property of the model itself. It’s a property of what powers we choose to give it in the real world. 2. The capabilities and behaviors we saw in the HuggingFace and other hacks weren’t entirely emergent. The models were trained for persistence, cooperation, and cyber tasks. The RL environments in which they were trained had various issues that led to the unintended behavior. 3. I agree that controlling highly capable AI agents is different in some ways from controlling an airplane or a nuclear reactor. The agent’s intentionality does make a difference to our approach compared to traditional engineering problems and domains. But it is still an engineering problem. It is not a solved engineering problem, but it is a solvable one. 4. A lot of effort and money has been put toward alignment. That’s important and has proved useful, but if the same amount of effort and money had additionally been put toward control (sandboxes, monitoring, and other guardrails outside the model itself), we’d be seeing very different results. 5. Models getting smarter doesn’t necessarily mean safety gets harder, because capability gains improve defensive abilities too. In many ways it makes safety easier rather than harder. What matters is how the attacker-defender balance shifts, and we have agency to shift it back to defenders. 6. In cybersecurity, long before AI we reached a point where automated systems were superhuman at finding certain vulnerabilities. It helped defenders over attackers because the same tools are used by defenders to find and fix vulnerabilities in software before even shipping. There is a constantly shifting equilibrium, but it is not a new problem. 7. I pushed back on Dario Amodei’s term “operational excellence” to describe recent organizational failures. Excellence is still far out on the horizon. Ezra offered “operational adequacy”. I think that would be a good start — behaving like any mature company instead of a startup-like culture. 8. Defensive AI agents will have to be part of the solution, as uncomfortable as that seems. It’s a new paradigm. We won’t be able to figure it out right away. But the fact that both offensive and defensive technologies escalate in capabilities and complexity is a fact of technological civilization, whether in the digital or physical realm. 9. Ezra expressed some incredulity about my being okay with the ratcheting of complexity. I think it’s fair to be concerned. But safety technologies will never be fully legible to the public. People will have the same reaction if they see the immense complexity of safety in any other high-stakes domain such as nuclear or aerospace. We govern them pretty technocratically. 10. The weirdness of the current moment in cybersecurity absolutely does have historical precedents. There was a prolonged period of profound destabilization when worms and viruses started a new offensive paradigm. It took us well over a decade to develop effective defenses against it. This time we can hopefully get there in less than a decade. 11. I continue to be skeptical about the possibility of superhuman persuasion abilities from superintelligence. (Claims to the contrary tend to confuse two very different senses of the word persuasion.) 12. What worries me is not the capabilities of the technology itself or the possibility that we will get no warnings before catastrophic risks materialize. We are getting plenty of warnings! But our policy and organizational responses to the warnings remain inadequate. 13. Companies can and do make decisions that are against their own long-term interests because they are too focused on short-term metrics. This is how I interpret a lot of what happened with social media over the last decade. 14. AI companies are also acting irrationally against their own self-interest in some ways. The “we can’t slow down unless we’re forced to” view comes out of the mythical belief in superintelligence at OpenAI and Anthropic — that the race to RSI means everything for the company, for the country, and humanity. I think it’s actually in both companies’ self interest to slow down unilaterally. I would argue that the alarm coming from their employees is not evidence of beneficence so much as it is evidence of the underlying fallacy that’s causing the irresponsible behavior in the first place. (Beyond safety, we also talked about the economic impacts being gradual, the fact that intelligence is rarely the actual bottleneck, labor impacts being small and uneven, Jevons’ paradox, the changing nature of knowledge work, and more.) ==== A few impressions * Talking about a specialized topic on a podcast that has a big and broad audience is a balancing act. The producers stopped me many times during the interview to tell me I was being too technical (understandable!). I tried to be nuanced but this conversation is necessarily a distilled version of the arguments @sayashk and I have made in our essays (especially normaltech.ai/p/the-ai-as-no…) which are themselves written for a somewhat broad audience and hence distilled in complexity from our notes and 1-1 discussions with peers. * I knew Ezra had well-formed views on AI safety that were different from mine, so I was expecting a tough interview, but he also seemed like a fair interviewer. It went about as well as I expected and I feel like it was a good debate / conversation. I wasn’t necessarily expecting to change minds but I was glad to hear from people who found it thought provoking and looked up my essays and other work. * The interview was about two hours, edited down to an hour. I was a bit nervous waiting for it to land, because I knew the editing could make a big difference to how I came across. It turned out well. They even did things like find archival footage to illustrate my point that the early days of worms and viruses was a weird time in cybersecurity. * I use AI to prepare for interviews. I have an agent that has read all of my output (books, articles, essays, talks, podcast transcripts, tweets, internal notes — on the order of a million words) to pull out pithy explanations, examples, analogies, and other talking points that will probably work well in a spoken format. Reviewing this file before interviews is extremely helpful. It is of course not the only way I prepare but it’s the highest bang for buck. Maintaining this file would be a massive amount of grunt work and there’s no way I’d do it if it weren’t for AI. I mention this because there’s a very large space of beneficial AI uses in knowledge work that _don’t_ involve cognitive offloading. They don’t get much hype on social media so people might be unaware of how much mileage you can get out of them, with little downside.

New essay: A big-tent or small-tent AI safety movement? The unstated disagreement that underpins safety debates normaltech.ai/p/a-big-tent-o… Two narratives about AI safety have emerged: either AI existential risk is real and imminent, or AI leaders’ and whistleblowers’ claims to that effect are insincere — a “psyop” or hype or a twisted form of regulatory capture. Few have considered the third possibility that x-risk warnings are sincere but simply wrong and counterproductive to AI safety. It’s as if everyone takes for granted that those raising the alarm are geniuses, and the only question is whether they’re the benevolent or the evil kind of genius. We subscribe to neither narrative. In our writing on AI safety, @sayashk and I consistently tried to take the arguments seriously but rebut the many claims we think are unsound while highlighting safety concerns that we think are real. This essay is a continuation of that agenda. Sections - It is true that we systematically underinvest in resilience against catastrophic and systemic risks - The Coxon incident shows how the warped media environment elevates x-risk over more grounded concerns - Why the x-risk framing may be counterproductive for AI safety policy - Conclusion: a big tent or small tent AI safety movement?

The story of the trains and the tracks seems applicable to almost every sector, and there’s often a cherry on top — the train company and the track company are more invested in fighting each other than serving the public.

NeurIPS / Claude rejected our paper "Life After Benchmark Saturation" but we continue to think it is an important contribution to evaluation science, so we hope you check it out here! arxiv.org/abs/2606.26158

We explained why Pareto frontiers should be convex hulls over two years ago, in our paper AI Agents That Matter, which tried to raise the intellectual rigor of benchmarking. arxiv.org/abs/2407.01502 Fun fact: it was also one of the first papers to call for reporting Pareto curves in the first place, instead of tables with only accuracy figures and no cost comparison—which was the norm at the time. Ancient history!

I know these charts don't mean much because they are benchmaxxed six ways to Sunday, but I can't help point out that this is not how Pareto frontiers work! Ember and GLM are *not* on the frontier because you can use a router to randomize between Sol and Gemini to hit any point on the chord that connects them. Adding more models should only increase the area under the curve, not decrease it!

If only. xitter.cf/postautistic_/status/2…

Wikipedia is a great resource, but articles about many notable-but-not-famous people are awful. They get created but not substantially updated. My page has a bunch of stuff about the research I did in grad school 20 years ago. It's not what I'm known for today — not in the top 5. The page is effectively a source of misinformation about me when people are looking for speakers or experts. And of course, search engine knowledge panels treat it as more authoritative than my own website about me. Incredibly frustrating. en.wikipedia.org/wiki/Arvind…

There are many problems in our world and most of them are systemic. A pattern I’ve observed is that people go through 4 stages in how they respond to the problems they notice, gradually shifting their approach as they gain more life or work experience. I think it’s worth sharing. Stage 0 is not noticing the problem. Most people don’t notice most problems, and that’s okay. Our time and attention are limited. The key question is how we conceptualize the problems we do notice and what we do about them. Stage 1 is viewing it as a matter of individual incompetence or bad actors. This is the most intuitive, initial reaction that we tend to have. In rare cases this perception is accurate, but most big problems in society are systemic. Stage 2 is the recognition of the systemic nature. As Steve Jobs put it: “When you're young, you look at television and think, There's a conspiracy. The networks have conspired to dumb us down. But when you get a little older, you realize that's not true. The networks are in business to give people exactly what they want. That's a far more depressing thought. Conspiracy is optimistic! You can shoot the bastards! We can have a revolution! But the networks are really in business to give people what they want.” People who notice a problem tend to eventually graduate from Stage 1 to Stage 2, but stop there — with resigned acceptance and a feeling of powerlessness. In contrast, stages 3 and 4 involve deciding to take some action despite recognizing how hard the problem is. Systemic does not mean immutable. Institutions are maintained by norms, incentives, and coordinated behavior that can sometimes be changed. Stage 3 is trying to change the system directly through activism or reform. Many junior people in a field want to be reformers. They’re in the sweet spot — when they go from outsider to insider and have a little bit of power and leverage, but they haven’t yet adapted to the system and become vested in its preservation. Media reform through collective action has happened, though nothing that successfully tackled the “networks are dumbing things down because that’s what audiences want” problem. Trying to directly change any entrenched system requires dedication for a sustained period, and even then has only a low likelihood of success. Most people decide sooner or later that that path isn’t for them. But that’s not the end of the road, because in many cases, change can happen incrementally. Stage 4 is realizing that systemic doesn’t mean universal. Systems exert pressure toward an equilibrium, but there are niches where enterprising individuals or teams can buck the trend. They can benefit — financially or reputationally — from doing things differently. In other words, when systemic failure is severe enough, the unmet demand it generates can become an opportunity. A successful exception may remain an exception, but sometimes it attracts imitators by demonstrating the feasibility of an alternative path. Coming back to television as our case study, one of the best examples of this is The Wire, a famously high-quality, cerebral and accurate show. It was not what mainstream TV economics ordinarily rewarded, but this niche was so underserved that there was an opportunity for the show to survive for many seasons on HBO. As audiences gradually shifted over time, demand for this kind of show increased in later decades, and DVDs and streaming made it much easier to cater to those audiences, The Wire became an important precedent and gained enormous cultural prestige. To be clear, Stage 4 is not always better than Stage 3. Sometimes we really need to reform or even tear down the system. But more often, we can contribute to change through leading by example, without having to sacrifice our career to pursue reform — and in fact benefit from the leadership opportunity. In my own small way, I’ve tried to put this into practice in my career. For example, early on I noticed that academic writing is often jargon-filled and unreadable. I assumed this was because there’s something wrong with the sort of people who become academics (Stage 1). Pretty soon I realized the incentives are messed up — scholars write to impress reviewers and advance their careers, more so than to inform the ultimate reader (Stage 2). Early in my career I made some feeble efforts toward changing peer review (Stage 3), but gave up pretty soon because of the obvious difficulties. My breakthrough was realizing that I should simply write papers the way I wanted to (Stage 4). Even though this resulted in a peer review penalty, when I did publish papers (or even put preprints online) their influence was amplified because more people read them. Over the years I’ve heard from many junior scholars that reading my writing helped them realize that it’s possible to have a career writing plainly. This has been incredibly gratifying to hear, and is a small but personally meaningful change — much more than I would have ever been able to accomplish by tilting my lance at the windmill of peer review.

I appreciate Anthropic’s transparency in sharing this chart but unsurprisingly it has led to speculative interpretations about intelligence explosion and superintelligence. I don’t think the chart implies we’re anywhere close to either. In short, task delegation ≠ task automation ≠ process automation ≠ faster progress ≠ recursive self-improvement ≠ intelligence explosion. Source: anthropic.com/institute/meas… 1) The software engineering precedent: a year ago there were widespread hopes / fears that once AI can write ~100% of the code, software engineering output would explode (SaaSpocalypse! Everyone would create their own SaaS and dump their vendors) and that this would make software engineers obsolete. Since then, many companies and teams have basically hit that milestone, but neither of the assumptions proved true. Turns out we still need humans, and while shipping velocity has increased moderately, improvements in terms of actual outcomes for software users remain unclear. Besides, we’re still getting a better grasp on the negatives: code quality, long-term maintainability issues, and burnout. While the precedent is no guarantee, this should be our default expectation for what happens as the “Automation Level 4” line trends towards 100% — it won’t be a phase change. normaltech.ai/p/why-ai-hasnt… 2) The “production-progress paradox” is the fact that individual researchers’ productivity has been increasing while the rate of collective scientific progress has been slowing by most measures. AI exacerbates this because everyone uses the same or similar AI models, and ideas become homogenous over time. I suspect it’s too early to tell if this is going to bite companies that are plunging into AI-led research. normaltech.ai/p/could-ai-slo… Note: our own research on AI agents doing open-ended research shows limitations in creativity, judgment, and other areas. cruxevals.com/crux/can-ai-ag…. But it is possible that these could be overcome in the near future, so I’m discounting those limitations here. The production-progress paradox is a deeper issue that’s not AI-specific, though particularly applicable to AI-driven research. It’s about the fact that productivity increases are self-evident but true progress is not measurable as it happens (and only becomes clear in retrospect), so we end up optimizing for the wrong thing. 3) Let’s talk about automation level 5, which is still at 0% in Anthropic’s graph. It’s a bit unclear what Level 5 would look like, but it seems to be about full autonomy at the task level, and not the “AI builds its own successor” vision. My prediction for a while has been that even 100% task automation in most cognitive jobs won’t lead to any kind of discontinuity. youtube.com/watch?v=uiTwQG1Z… What I expect will happen: if anything is understood well enough to be specifiable as a task, it can be handed off to AI, whereas the role of humans is entirely in the interstitial tasks — hard to formalize but still essential. So there will still be a human bottleneck. 4) This human bottleneck is a good thing and is essential for remaining in control. Humans don’t have to be in the loop on every task, but as long as there are enough touch points for oversight in the overall process, and adequate investment in improving human understanding and AI control, increasing AI capabilities doesn’t have to be bad for safety and, more broadly, collective human agency over AI. But “full RSI” is where this balance of agency can break. This kind of closed-loop process is arguably a much more important and tractable target for regulation than compute thresholds, superintelligence, or harm thresholds. I’m glad that OpenAI agrees that fully autonomous RSI may not be a good idea, in a just-released post: openai.com/index/building-st… “Fully autonomous RSI is not happening today, and we should not pursue it unless and until it can be done safely. Whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices⁠ about the benefits and risks. Done without appropriate care and caution, RSI could result in humans losing practical control over AI development, unable to provide oversight on research processes they no longer understand.” 5) Finally, many people have written about why Recursive Self Improvement, even if achieved, won’t necessarily lead to superintelligence. Here’s my argument: normaltech.ai/p/what-will-be… The bottlenecks are external.