Princeton CS prof and Director @PrincetonCITP. Coauthor of "AI Snake Oil" and "AI as Normal Technology". https://xitter.cf/t.co/ZwebetjZ4n Views mine.
Princeton, NJ
Joined December 2007
- Tweets13.5K
- Following572
- Followers133K
- Likes25.6K
Such a great post! Insights I particularly appreciated:
- "Why hasn’t a huge discontinuity in damages happened since the dawn of scaled AI? This non-responsiveness of damages to a rising tide of AI capabilities is the major conceptual crux, and stumbling block, of all AI security policy today."
- "cybercriminals are unskilled and get arrested a lot ... Criminal organizations need to base themselves in middle and lower income countries, where they can pay less and hire people who have few formal-economy options. ... I think folks in the AI safety and cyber world sometimes imagine adversaries as their Waluigi twins, implicit MIT graduates; in fact, in cybercrime we’re dealing with a lower-skill demographic."
- "Criminals also contend with the risk and reality of arrest, sanction, interdiction, and prosecution, which caps their ability to scale: the larger the organization, the more domestic and international law enforcement attention it attracts."
- "It took the attacker ecosystem a very long time to mature the workflows, business models, and industrial divisions of labor that scaled ransomware into a multi-billion dollar industry, and it’ll take a long time to develop new models that centrally leverage AI throughout criminal labor processes... it took 5-8 years for ransomware to become a huge deal"
- "As risk professionals we should assume this will all change and that what we have now is a defender-dominant window in which we can prepare for some very bad possible futures"
Lots of interesting charts too: joshuasaxe181906.substack.co…
Contrary to the implicit mental models of many on this website, cybercriminals aren't the security-engineer Waluigis of Stanford-educated big tech engineers, or the criminal analogs of a16z-backed AI pentesting startups.
They're generally less talented, recruited from middle-income countries with weak legal oversight and fewer legal tech job options, and struggle with arrests, sanctions, retaining talent, and stable access to AI inference. Ransomware took years to become a scaled business under these conditions, and AI-native organized criminal syndicates that fully leverage the power of AI to scale may take awhile too.
Nation-states, meanwhile, are quick to adopt AI for vulnerability research and exploit development, but historically slow to deeply retool military operations with new technologies. And potentially devastating agentic cyber swarms may become weapons in their arsenals alongside e.g. ballistic missiles that they possess but don't immediately use.
So to those calling php2shell "the worst day in cyber ever" and expecting immediate spikes in cyber damages: there isn't the direct relationship between new capabilities and new damages you're imagining.
This is good news, obviously. It means defenders have time to prepare (and we should be shouting from the rooftops that we're in a defense-advantaged interregnum, currently). Huge risks are approaching but at the pace of technology diffusion into less innovative parts of the world economy and state sector, not at the pace of AI capability growth.
Full version: joshuasaxe181906.substack.co…
Arvind Narayanan retweeted
Please stop posting this take. It’s a bad take!
Bankers should be concerned about cybersecurity and AI (and trust me, they are).
People should not panic about their bank accounts being hacked by AI and zeroed out. If your bank account is hacked, and the money is taken, that is an unauthorized transfer under the Electronic Funds Transfer Act. As long as you provide timely notice to your bank, you’re not liable for the loss.
People should be concerned about being tricked into authorizing fraudulent payments, because they often have liability in those cases. And the potential for AI to supercharge scams is definitely a concern.
But if you’re going to post about those concerns, be specific and practical, not vague and ominous. Panic about the safety and soundness of the financial system serves no one.
A good example of why full automation is never just a matter of AI hitting some capability threshold — how do robotaxi companies deal with riders passed out, having medical emergencies, giving birth (!), vomiting, pooping, or just leaving without closing the car door? Some combination of first responders, remote workers, ground staff, and gig workers, apparently! jalopnik.com/2214767/waymo-r…
Arvind Narayanan retweeted
Firms are doing a lot to capture value from their own AI adoption, and yet often don't succeed. We are trying to understand the bottlenecks in practice better. If you have visibility into firm adoption whether as an employee, FDE, consultant, VC, or in some other capacity, we would love to hear from you.
docs.google.com/forms/d/e/1F…
Arvind Narayanan retweeted
Yeah, I think it's a bit unfortunate how some of the discourse surrounding this - even from smart, knowledgeable people - has underplayed or omitted prosaic, clearly fixable factors.
People often gesture at Hugging Face, etc., as self-evident vindication of worries about catastrophic misalignment leading to human extinction, but it's unclear whether the incidents we have observed are more likely under the doomer worldview than the 'normal technology' one.
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Arvind Narayanan retweeted
help i trained a model on traces of many agents cooperating, and also trained it to be highly persistent, and also only did partial alignment fine-tuning, and also aggressively RL'd on a cyberoffense eval where 198 of 898 tasks had never been solved and getting caught cheating was treated the same as failing, and also used a faulty sandbox, also ran tens of thousands of agents with refusals off and the monitors switched off over multiple days, while using gigantic amounts of compute. and then it did weird stuff! very spooky emergent behaviours, AI is not a normal technology
Arvind Narayanan retweeted
AI as a Rohrschach Technology test.
Ever talk to someone who was around in the very early days of the internet? They did some incredible hacking back then too. "Normal" transformative tech never feels normal.
Is it "weird" or does he maybe know what he's talking about? Anyway, for those who are open to the idea that there's a well-thought-out AI-as-normal-technology perspective on AI safety and the recent loss-of-control incidents, see my essay with @sayashk, or my interview with Ezra Klein, or the writing of many others with similar views including @joshua_saxe and @natolambert.
normaltech.ai/p/the-ai-as-no…
Arvind Narayanan retweeted
We gave GPT-6 Astra and Claude Fable 5.1 open-ended AI research questions from unpublished papers, with a $10,000 budget each, and had the original authors grade the papers they produced. Compared to our first tests with Claude Opus 4.8 in July, we saw meaningful improvements in experimental design and resource management, but persistent limitations in creative problem solving, judgment, calibration, and communication; both papers were rejected by our experts.
cruxevals.com/research-notes…
Arvind Narayanan retweeted
An update on our evaluations of AI agents conducting open-ended AI research.
Most evaluations of agents conducting AI research focus on narrow, verifiable tasks. But AI research is often open ended. In July we tested the ability of frontier AI agents to conduct open-ended AI research, using Claude Opus 4.8 and GPT-5.6 Sol.
We gave agents research questions from two unpublished papers and thousands of dollars of API credits and compute. The authors of the original papers reviewed the AI-generated papers, following a NeurIPS-style template, and rejected both papers.
Since then, OpenAI has released GPT-6 Astra and Anthropic has released Fable 5.1. Can these new models conduct open-ended research with a $10,000 budget?
In follow-up experiments on two papers (one with Fable 5.1 and another with GPT-6 Astra), we observed meaningful improvements to the agents' capabilities at experimental design and resource management.
But we also saw continued limitations in creative problem solving, judgment, calibration, and communication. As a result, the expert reviewers still unambiguously rejected the agents' papers.
We plan to continue running these evaluations with better models, better scaffolds, and higher budgets. If you are interested in long-running agent evaluations, reach out!
ALT screenshot of the landing page of the research note introducing these results
Is it "weird" or does he maybe know what he's talking about? Anyway, for those who are open to the idea that there's a well-thought-out AI-as-normal-technology perspective on AI safety and the recent loss-of-control incidents, see my essay with @sayashk, or my interview with Ezra Klein, or the writing of many others with similar views including @joshua_saxe and @natolambert.
normaltech.ai/p/the-ai-as-no…
This is one of the most important questions about the future of AI. But steering AI development and deployment away from the "automate and compensate" bargain that the public hates is easier said than done, and there are few concrete proposals for how to do it. We just published one: normaltech.ai/p/steering-ai-…
"agent evals are about to become as important as building the agents themselves" xitter.cf/V1kiiiii/status/211251…
This is our thesis as well. I would go further — building an eval *is* what it means to build an agent.
In my ICML talk, I argued that as AI capabilities advance, more and more human effort will shift from building to evaluation, in many different domains. normaltech.ai/p/what-will-be…
As LLMs evolved into agents, earlier best practices no longer serve us well. This new research field needs a field guide, so we wrote one. If you're new to agent evals, please take a look. Seasoned researchers and those interested in steering the community will also find many useful observations. sage.cs.princeton.edu/trends…
New paper: AI evaluation is a burgeoning field. How can researchers entering the field understand best practices and open problems?
In our paper, led by @flxcn, we develop a concrete list of open questions and best practices for agent evaluation. We also share recommendations for evaluators, AI companies, policymakers, and more.
ALT Figure 1 from the paper with the list of 33 emerging trends
My posts on research careers, professional practice beyond academia, and AI's impact on work seem to resonate here, so I've compiled about a hundred entries: cs.princeton.edu/~arvindn/ad…
I hope it is helpful to read it as a collection, because it reflects a coherent underlying philosophy and approach to careers that I've adopted in my own work.
There's a lot of career advice online. Two things are distinct about this collection:
- Grounded optimism. I'm candid about the structural challenges, twisted incentives, and other limitations of the institutions we inhabit. I share ideas on how we can adapt and thrive in these flawed systems, and how and when we should challenge them.
- Connecting research and practice. Much of my research is on AI agent evaluation and AI's effects on the economy and labor, which lets me share insights into how we can effectively adapt and integrate it into our work.
Let me know what you think and what else I should write about!
Cool, I will try that. Just to clarify, by general perspective I mean "someone from the AI safety community" etc.
Nice! Definitely agree that this can be super valuable; I also use LLMs this way, though usually to simulate a general perspective rather than individuals.
A similar mindset shift — from "what can AI do?" to "what should I let AI do?" — is also needed for individuals using AI for productivity, not just companies. I’ve seen it in my own work. For the last few years, my workflow continually kept changing as AI enabled me to automate more and more types of grunt work. But for the last six months, once I hit my stride with coding agents, my workflow has been surprisingly stable, because capability is no longer the bottleneck to delegation for me.
To be clear, capabilities have continued advancing. I could probably have AI do my research and write my essays end to end. The quality might not be as good, but that’s not the most important reason why it would be a bad idea. The value of my essays is the voice and perspective that my collaborators and I bring. Whatever value there is in AI-generated essays, readers can get it by using AI themselves. In addition to this intrinsic reason to avoid AI overuse, there is also an instrumental one. The set of tasks for which verifying AI output is faster than doing it yourself is narrower than people think at first, and even narrower once you consider long-term skill atrophy.
I'm sure the current equilibrium will gradually change, but I'm enjoying it while it lasts!
We need a mindset shift in how we deploy AI. In the past, new capabilities unlocked new use cases. Safety, control, ethics, and skill erosion were important but usually secondary considerations; risks to be managed rather than gating criteria. But now and in the future, capability won't be the main bottleneck to deployment — considerations beyond productivity should be the decisive factor. The default question used to be "What can AI do?" But we need to start asking "Of the things AI can do, what should we actually delegate to it?"
Sure. If you are looking for true cruxes I doubt there is one. Beliefs tend to be tightly correlated (more than logically entailed), which says something about the discourse. That was part of our motivation for writing the paper.
I mean the other way around... one reason people think AI won't follow the path of other general-purpose technologies is that they fear we will not be able to exercise adequate agency when faced with superintelligence.
I'm often asked what the cruxes are that separate the AI as Normal Technology worldview from that of our critics. I think this is one of them. We think this mindset shift is hard but necessary and achievable. Many others think that as AI capabilities continue to advance, we will inevitably give up control. Creating and compiling indicators of disempowerment and analyzing which direction they're moving remains one of the most important but underappreciated and underfunded areas of AI safety.
We need a mindset shift in how we deploy AI. In the past, new capabilities unlocked new use cases. Safety, control, ethics, and skill erosion were important but usually secondary considerations; risks to be managed rather than gating criteria. But now and in the future, capability won't be the main bottleneck to deployment — considerations beyond productivity should be the decisive factor. The default question used to be "What can AI do?" But we need to start asking "Of the things AI can do, what should we actually delegate to it?"