Building something new. Previously: CTO at Autonomy
San Francisco Bay Area
Joined November 2007
- Tweets18.8K
- Following1.6K
- Followers4.3K
- Likes6.2K
mrinal retweeted
On the upside: The fact that old bugs are being discovered is a strong indicator for optimism: Old bugs are deep bugs that survived many years of code review. Finding a bunch of old bugs is the clearest indicator that technological progress in bug detecdtion is real.
mrinal retweeted
The biggest misread of anyone predicting the end of software engineering was imagining that the world only needed a certain amount of software, and bringing automation to the field would reduce the need for labor.
Exactly the opposite is happening where essentially most things are actually turning into software. Because agents can use tools, work with computers, write software on the fly, and process data like humans (but far faster), we can now throw “software” at problems that never could reasonably be digitized before.
What that leads to is the role of engineering taking on greater importance in the economy, because we can now engineer far more systems and processes than were possible before.
If you’re a bank trying to do new forms of trading, engineers are often your first bet. If you are in life sciences, you expect more and more research and lab work to be engineering centric. If you’re in manufacturing, you’re trying to automate more of the supply chain, all that is engineering. More and more work is shaped like software problems.
Yes we’re not going to be coding manually in the future. But almost everything will be code, which means those that understand it are in a fantastic position.
Some observations about the "total" automation of software engineering:
1. There are more software engineers now than in 2025, and demand is growing.
2. Software engineers have never worked harder than they do now.
3. They largely don't write code anymore, agents write ~100% of the code.
4. The bottom line of the companies that employs these engineers and agents (i.e. the terminal purpose of all this code) is typically not budging as a result of AI adoption.
This is why you have to think in terms of systems. Automating one aspect of a system creates its own friction and moves the bottlenecks elsewhere.
mrinal retweeted
Ok, this is cool. Someone rigged up SlateDB as an extension for DuckDB. 🤯
dentiny.github.io/posts/duck…
mrinal retweeted
Here are 14 points about AI safety I made on the Ezra Klein show, and some impressions about the experience. nytimes.com/2026/10/09/opini…
1. Intelligence isn’t power — An A.I. system being powerful is not a property of the model itself. It’s a property of what powers we choose to give it in the real world.
2. The capabilities and behaviors we saw in the HuggingFace and other hacks weren’t entirely emergent. The models were trained for persistence, cooperation, and cyber tasks. The RL environments in which they were trained had various issues that led to the unintended behavior.
3. I agree that controlling highly capable AI agents is different in some ways from controlling an airplane or a nuclear reactor. The agent’s intentionality does make a difference to our approach compared to traditional engineering problems and domains. But it is still an engineering problem. It is not a solved engineering problem, but it is a solvable one.
4. A lot of effort and money has been put toward alignment. That’s important and has proved useful, but if the same amount of effort and money had additionally been put toward control (sandboxes, monitoring, and other guardrails outside the model itself), we’d be seeing very different results.
5. Models getting smarter doesn’t necessarily mean safety gets harder, because capability gains improve defensive abilities too. In many ways it makes safety easier rather than harder. What matters is how the attacker-defender balance shifts, and we have agency to shift it back to defenders.
6. In cybersecurity, long before AI we reached a point where automated systems were superhuman at finding certain vulnerabilities. It helped defenders over attackers because the same tools are used by defenders to find and fix vulnerabilities in software before even shipping. There is a constantly shifting equilibrium, but it is not a new problem.
7. I pushed back on Dario Amodei’s term “operational excellence” to describe recent organizational failures. Excellence is still far out on the horizon. Ezra offered “operational adequacy”. I think that would be a good start — behaving like any mature company instead of a startup-like culture.
8. Defensive AI agents will have to be part of the solution, as uncomfortable as that seems. It’s a new paradigm. We won’t be able to figure it out right away. But the fact that both offensive and defensive technologies escalate in capabilities and complexity is a fact of technological civilization, whether in the digital or physical realm.
9. Ezra expressed some incredulity about my being okay with the ratcheting of complexity. I think it’s fair to be concerned. But safety technologies will never be fully legible to the public. People will have the same reaction if they see the immense complexity of safety in any other high-stakes domain such as nuclear or aerospace. We govern them pretty technocratically.
10. The weirdness of the current moment in cybersecurity absolutely does have historical precedents. There was a prolonged period of profound destabilization when worms and viruses started a new offensive paradigm. It took us well over a decade to develop effective defenses against it. This time we can hopefully get there in less than a decade.
11. I continue to be skeptical about the possibility of superhuman persuasion abilities from superintelligence. (Claims to the contrary tend to confuse two very different senses of the word persuasion.)
12. What worries me is not the capabilities of the technology itself or the possibility that we will get no warnings before catastrophic risks materialize. We are getting plenty of warnings! But our policy and organizational responses to the warnings remain inadequate.
13. Companies can and do make decisions that are against their own long-term interests because they are too focused on short-term metrics. This is how I interpret a lot of what happened with social media over the last decade.
14. AI companies are also acting irrationally against their own self-interest in some ways. The “we can’t slow down unless we’re forced to” view comes out of the mythical belief in superintelligence at OpenAI and Anthropic — that the race to RSI means everything for the company, for the country, and humanity. I think it’s actually in both companies’ self interest to slow down unilaterally. I would argue that the alarm coming from their employees is not evidence of beneficence so much as it is evidence of the underlying fallacy that’s causing the irresponsible behavior in the first place.
(Beyond safety, we also talked about the economic impacts being gradual, the fact that intelligence is rarely the actual bottleneck, labor impacts being small and uneven, Jevons’ paradox, the changing nature of knowledge work, and more.)
====
A few impressions
* Talking about a specialized topic on a podcast that has a big and broad audience is a balancing act. The producers stopped me many times during the interview to tell me I was being too technical (understandable!). I tried to be nuanced but this conversation is necessarily a distilled version of the arguments @sayashk and I have made in our essays (especially normaltech.ai/p/the-ai-as-no…) which are themselves written for a somewhat broad audience and hence distilled in complexity from our notes and 1-1 discussions with peers.
* I knew Ezra had well-formed views on AI safety that were different from mine, so I was expecting a tough interview, but he also seemed like a fair interviewer. It went about as well as I expected and I feel like it was a good debate / conversation. I wasn’t necessarily expecting to change minds but I was glad to hear from people who found it thought provoking and looked up my essays and other work.
* The interview was about two hours, edited down to an hour. I was a bit nervous waiting for it to land, because I knew the editing could make a big difference to how I came across. It turned out well. They even did things like find archival footage to illustrate my point that the early days of worms and viruses was a weird time in cybersecurity.
* I use AI to prepare for interviews. I have an agent that has read all of my output (books, articles, essays, talks, podcast transcripts, tweets, internal notes — on the order of a million words) to pull out pithy explanations, examples, analogies, and other talking points that will probably work well in a spoken format. Reviewing this file before interviews is extremely helpful. It is of course not the only way I prepare but it’s the highest bang for buck. Maintaining this file would be a massive amount of grunt work and there’s no way I’d do it if it weren’t for AI. I mention this because there’s a very large space of beneficial AI uses in knowledge work that _don’t_ involve cognitive offloading. They don’t get much hype on social media so people might be unaware of how much mileage you can get out of them, with little downside.
mrinal retweeted
And specifically for cryptanalysts who are working with models and for the labs:
Folks, what we need right now are *negative* results. Silence is the worst thing you could be publishing.
mrinal retweeted
the jev Decision Index v0.3 is here 🎯
featuring a way stronger set of benchmarks, now with a strong private half 🔒 so the ranking tracks skills beyond benchmarks
and, as promised, vision 👁️
huggingface.co/spaces/multim…
mrinal retweeted
First question on AI.
Is the problem superintelligence or distorted intelligence?
This is a question I tried to address in this piece. I can add the following additional comment here.
I believe the debate is being obfuscated by equating goal-driven, autonomous behavior with intelligence. Plants and insects have goal-driven, largely autonomous behaviour. But this isn’t what we mean by intelligence in general.
Intelligence is general-purpose and holistic, and is typically well-adapted to its environment.
There is no doubt that current AI models have impressive capabilities. But in attempting to achieve these capabilities, AI labs may be creating distorted intelligence rather than superintelligence, as I explain in this article.
This matters because the dangers we should watch out for, the kinds of policies and regulations we should adopt, and the hope we should hold depend on the answer to this question.
Recent security breaches suggest that AI capabilities are developing rapidly while being put in the service of imperfect quantitative metrics that ultimately distort the behavior of the most advanced models, @DAcemogluMIT writes. project-syndicate.org/commen…
mrinal retweeted
This was a remarkably contentious thread, particularly among people who don't understand what an LLM actually is.
So let me follow up with two points about consciousness, memory, and individuality in AI.
First: where exactly does consciousness enter the training process?
If you've ever trained a small language model, you know what the beginning looks like. A tiny model produces garbage. Increase parameters, training data, compute, and training quality, and the model becomes progressively better at predicting what comes next.
Scale that far enough and the results become remarkably human-like.
But the underlying process is continuous optimization. Loss decreases. Weights change. Predictions improve.
So where is the transition from statistical prediction to subjective experience? At what point during training does the poorly performing model become aware?
There is no demonstrated transition.
What we can demonstrate is increasing capability and increasingly convincing human-like behavior. Calling the latter consciousness doesn't demonstrate the former.
And this is where I think some of the philosophical arguments get ahead of the evidence.
Understanding or debating the definition of consciousness does not give us a mechanism for detecting subjective experience in an LLM. We can define consciousness, agency, self-awareness, qualia, or any other philosophical concept as carefully as we want, but definitions aren't evidence.
And when we look at what is actually happening inside these systems, we can explain the apparent memory, identity, self-reflection, and continuity mechanically. None of it requires consciousness to explain. What we can demonstrate is an increasingly convincing emulation of conscious behavior, not consciousness itself.
How do we know? Because we designed and built the system. We know how it works. We can explain the behavior without invoking consciousness.
Second: memory and individuality.
Once trained, an LLM's weights are fixed during inference. The model doesn't sit around contemplating things between requests to make itself better. An inference server receives input, performs inference using the model, returns output, and then waits for another request.
No inference, no activity within the model. Between inference runs, the model isn't doing anything.
What looks like persistent memory, identity, continuity, and in-conversation "learning" comes from context provided to the model via API.
Conversation history, system prompts, tool definitions and results, retrieved memories, and other state are assembled by software around the model, formatted and tokenized, and supplied as context for the next inference.
Change that context and you change what the model "remembers." Delete it and the apparent continuity disappears. Compact or summarize it and details disappear or change.
The model didn't forget anything; the input changed.
And this creates a fascinating problem for claims of AI individuality.
If that context constitutes the AI's memory, and its continuous memory constitutes an individual conscious existence, then deleting a context should amount to destroying that individual.
Compact it and you've altered its memories.
Fork it and you've created two individuals.
Copy it to another instance of the same model and you've moved the individual.
Edit it and you've rewritten its past.
Delete it, and you've murdered it.
How dare you!
Or, much more straightforwardly, we are manipulating externally managed state that gives a statistical model the appearance of persistent memory and identity.
The model has not maintained a continuously evolving internal state between those calls. The surrounding system maintained state and gave it back to the model.
This distinction is going to become increasingly important because AI will become extraordinarily good at appearing conscious. Persistent memory, self-reference, emotion, introspection, personality, and identity can all become increasingly convincing.
But demonstrating the behavior of consciousness is not the same thing as demonstrating subjective experience or consciousness.
A model of a self is not evidence of a self.
Memory supplied as context is not evidence of an experiencing or learning mind.
And increasingly convincing human behavior is not evidence that something inside is experiencing it.
The mechanical explanations for these behaviors are well understood. Calling the resulting behavior consciousness adds an extraordinary conclusion without demonstrating the thing being claimed.
mrinal retweeted
A bit delayed, but the Navier–Stokes results sparked a lot of conversations about privacy, and inspired me to write a primer on what AI labs actually do with our data. I go through @OpenAI , @AnthropicAI, @GoogleDeepMind , and @spacexai policies: training defaults, retention, feedback exceptions, and what deleting a chat actually does.
Putting the policy details aside, one thing about this whole debate is worth emphasizing: I think focusing only on whether someone read the chat logs misses something.
Sometimes the valuable secret is just one bit: someone has already made this approach work with an AI. You don’t need their proof, their name, or their conversations for that to change where you put your time and compute.
This is why I don’t think Clio-style aggregation and LLM privacy filters are enough. Hiding the raw conversations doesn’t settle what someone can learn.
Full blog below 👇
mrinal retweeted
If coding agents will write most of our code, what happens with our communities and sense of ergonomics? How does it impact our compilers and tools?
mrinal retweeted
I'm seeing a lot of euphoria about how Opus 5.5 is good at TLA+, and this means that all software will soon be formally verified. As a person who loves TLA+ so much he wrote a book on it, I want to throw a particular cold shower on people's enthusiasm by talking about the limits of what you can actually verified with it.
The high level simplification is that TLA+ sees a system as a set of "behaviors", or possible sequences of states. For example, the pseudocode "pick a random number from 1-3 and decrement it to 1" has three behaviors: `{3 -> 2 -> 1, 2 -> 1, 1}`. From here, there are two basic kinds of TLA+ properties:
- `[]P` means that `P` is true in *all states* of *every behavior*.
- `<>P` means that `P` is true in *at least one state* of *every behavior*.
`[]P` is immediately useful as an **invariant**, or something that always be true of your system. This is things like "your data is never corrupt" or "there's always at least one server online." `<>P` is a little more abstract, but for technical math reasons I won't get into here, can be stacked with `[]` to create really complex and useful properties. `<>[]P` represents things like "the algorithm eventually converges on the right answer", `[]<>P` things like "if two data stores desync, they will eventually resync", and `[](P => <>Q)` things like "If a message is put on the queue, it's eventually processed by a worker".
Really cool stuff!
These primitives were chosen to make a wide array of properties useful. And if we're clever, we can do all sorts of more complex properties, like bounded time constraints and history properties. But we're always constrained to 1) define a logical formula 2) over individual behaviors, and 3) check that all behaviors satisfy that formula.
So some things that we *cannot* express in TLA+:
- Possibility and reachability properties: that it's always possible to *make* P true, even if you don't actually decide to. Things like "I can always shut down the computer" or "A user can always change their password". These can't be expressed with `<>P` because that's "for all behaviors, P happens at least once", we actually want "for all behavior prefixes, there is at least one behavior where P happens at least once".
- Hyperproperties: properties that are defined over two or more traces. These are things like "painting a car red doesn't make it faster" or "users cannot infer secret data by observing public data". We can't do these because TLA+ only looks at one behavior at a time.
- Statistical properties: 95% latency is 1ms. Impossible because most of these are hyperproperties.
- Properties about if a system is robust against code changes. Impossible because, uh, you have new behaviors now.
Some of these are solvable in different logical formalisms. CTL can do reachability, PRISM can do statistical properties, etc. Those have their own tradeoffs and limitations, though, and no system can do everything. Others are solvable with a lot of cleverness tailored to the specific spec, like lifting a model into a hypermodel. But these are insanely inefficient and make your "clever spec" diverge significantly from the real world system, so introduce a lot more opportunity for things to go wrong.
The core problem, though, is (1): properties are logical formula. If we don't know how to express a system property as a logical formula, we can't verify it. 99% of the properties we care about fall under this. The information on the site is easy for a user to find. Our LLMs behave as we expect them to. Our application can't be used to break the law. TLA+ (and Quint and Lean and Rocq) are near-useless here, no matter how clever you are.
Don't get me wrong: `[]P` and `<>P` represent a huge range of useful properties and TLA+ is incredible at finding awful concurrency bugs. But there's a lot it fundamentally can't do and we shouldn't believe that it will solve all our worries about software bugs. And the same goes for all other formal verification languages, too.
mrinal retweeted
updated Decision Index 0.1 → 0.2 🎯
better formula, +29 jev-like models, +21 benchmarks
AutoJev-27B by @perplexity_ai CTO @denisyarats took the open lead, trailing jev by 0.8 points 🏆
come find the best model at every size, speed and use-case
huggingface.co/spaces/multim…
mrinal retweeted
The loudest voices stoking fears about AI dangers have made tremendous headway in the past two weeks. AI technology has not taken some unexpected, dangerous turn, but the hype around it — propelled by what appears to be a well orchestrated PR campaign — has drummed up considerable fear. I worry that it represents a setback for our field.
I have written frequently that fears of AI are overhyped. AI’s capabilities can be uncannily human-like and unpredictable, and it’s rational to worry when people who are directly involved express concerns. But I see the problems as a sign of the engineering work that ahead, rather than insurmountable barriers or the sky falling. AI technology continues to advance — which is a good thing! — but technical advances, poorly understood by the public, give those who seek to generate hype repeated opportunities to do so.
First, I don’t see any step up in the risk of human extinction from AI compared to a few months ago. The theories about this remain the same fantastical, science fiction scenarios as a few months ago. The biggest change in AI risk is its cybersecurity capabilities — a topic which we should take seriously — but this, too, will not lead to the end of the world.
The most notable recent event leading to increased fear was when an OpenAI team deployed an agent swarm that hacked into Hugging Face. Much of the popular press contained significant hype. For example, some publications reported that a swarm of 1,200 agents carried out the attack. While this was technically accurate, as I write this, I have about 1,300 processes running on my laptop. Yes, the ability to get large swarms of agents to work in parallel on a task is a significant technical advance, And, in computing, many processes run at the same time. So this shouldn’t be seen as some magical capability.
Additionally, OpenAI’s buggy sandboxing and monitoring processes were key to enabling this incident. Fixing these bugs and putting in place improved monitoring would be appropriate fixes, not pausing AI. There are many well known ways to attack software systems. The main advantage of AI agents is that they are relentless. They will tirelessly try many tactics — and have the patience to chain vulnerabilities together — that previously would have taken an infeasible amount of human effort. But in the long term, I believe the advantage will lie with defenders (because they have more information with which to identify bugs, which they can fix), but the cyber-threat landscape has changed significantly. There are still bottlenecks to identifying and exploiting a vulnerability. AI agents still have to try a lot of things to see what works, and taking these actions takes time and might be detected by defenders. This is why, even though it is now easy to obtain versions of leading open weight models that have had their guardrails removed or weakened, so they will not refuse to try to execute cyber attacks, the world has not ended.
I am also concerned about the anthropomorphization of AI in a lot of reporting, where LLMs and agents are unnecessarily treated as if they were people. If I wield a hammer, miss a nail, and accidentally dent the wall, it’s not the fault of the hammer. The problem lies in how I used the hammer. Similarly, if I prompt an agent and it hacks into someone else’s system, the responsibility lies with me, not the agent.
Of course, we want to build systems that are as safe and predictable as possible. (For example, an unsafe hammer would be one whose head randomly flies off under normal use.) Today’s agentic systems are not predictable, but I see no reason why, by applying sound engineering practices, we won’t be able to make them extremely safe to use. One new element in the forecasts of AI-enabled doom is AI companies disclaiming responsibility for their own products. “I didn’t do it; my out-of-control agent did!” There’s a balance to be struck between the responsibility of the tool maker and the tool user, but when something goes wrong, let’s hold the people building and/or using the hammer responsible, rather than the hammer. (By the way, if you’re worried about AI bioweapon risk, David Bellamy has a great post on why this, too, is overhyped. Briefly, the bottleneck in building a bioweapon is not intelligence, but lab work and manufacturing.)
Pausing AI progress will create much more harm than benefit. First, our adversaries will certainly not slow down. Second, engineering requires discovering problems empirically so we can fix them. If we pause AI by a decade, we will also delay finding and implementing safety engineering fixes by about the same duration.
Of course, the incentive to stoke fears — for regulatory capture, to garner attention, or to make one’s technology seem more powerful — remains the same as before. Disclaiming responsibility is a new one. Taking a hard technical look at the actual risks however, I see little factual basis for the degree of fear that’s been stoked up. We still have hard research and engineering work ahead to improve AI safety, but the beneficial applications continue to vastly outweigh the risks, and we should keep building.
[Original text (with links): deeplearning.ai/the-batch/is… ]
mrinal retweeted
If you *think* your pushes to GitHub are protected by your SSH key being on a yubikey, double-check that you don't have valid github auth tokens sitting in your home directory:
$ gh auth status
Embarrassing story: for a decade I thought my git push to github was protected by my yubikey tap.
Then one day an agent that couldn't get me to tap because I was away noticed that I had the gh command installed and it allows access to a github token that can be used to push via the API 🤦♂️
@dinodaizovi is right of course. I love how good agents can be at helping you harden security boundaries if you focus them on that problem.
mrinal retweeted
Kev-0.5B: A tiny open source Jev-like decision model with a TypeSafe-compatible API based on Qwen2.5-0.5B that you can train and run on a MacBook Pro.
Model card and weights are available on GitHub
github.com/jaredpalmer/kev
mrinal retweeted
I ran some real, live evals on Jev vs DiffusionGemma-as-Jev (my patch for vLLM!)
DiffusionGemma comes out as the winner, I think.
Headlines:
Is Jev faster than DiffusionGemma? No ❌ (API vs DGX Spark)
Is Jev smarter than DiffusionGemma? No ❌ (they're roughly tied!)
mrinal retweeted
It's frustrating that the discussion about safety and pacing of AI progress is once again led by a handful of people when the implications concern everybody.
Let me lay out a constructive, alternative proposal. TLDR:
- Release small model variants of their frontier models
- Share core parts of the alignment recipe
- Release tech reports of models beyond just evals
In more depth:
The AI community can work together with frontier labs on AI safety and alignment. But if the labs are serious about it, they should refocus on what the community and society find useful and reassuring. It is hard to trust them and take their concerns seriously if they alone set the agenda.
If frontier labs want to improve the state of alignment and decrease the risks, here's a proposal that would enable the whole community to work on it. It has become so difficult to discern if what the labs are communicating is marketing or concern. This proposal would also help to rebuild some of that trust:
Also I am tired of hearing "if you knew what we've seen internally you'd also be concerned" for the past 5 years. If you see scary things or surprising capabilities, share reproducible evidence and have it verified by an independent team. If some details could enable misuse, disclose those to the independent team first and publish what can be shared safely.
So are these three points important?
**Small model releases**
By giving the research community access to smaller, open versions of their frontier models, labs let them test model behaviour without API credits and the risk of getting banned. Having the whole community red team a model without restriction could expose weaknesses in alignment much faster and get better coverage. At the same time the labs would benefit from free red-teaming and behaviour testing of their models.
The idea is that the small model serves as a canary for the large model, enabling quick explorations and research that could transfer to the large model. Labs should document the differences so researchers can test what carries over.
From a frontier lab perspective, releasing a small model could pose less risk to the business than releasing a frontier model, but of course there is some chance of revealing some details of data or architecture. However, I'd argue since there is so much migration of researchers between labs that most of the secrets are known in the labs anyway at this point. I'd like labs to be specific about which details they can't share and why.
**Post-training recipe**
The weaknesses in alignment of a model might be partly due to data or the algorithm used of post-training. Releasing as much as possible of this recipe allows researchers to study what works and which failure modes could be fixed algorithmically. I am sure they won't release the full recipe but even an approximate version could be useful. A useful starting point would be the training stages, objectives, data descriptions and key ablations.
**Tech reports**
The tech reports of the early days (e.g. GPT-2 or GPT-3) have led to some transparency what the frontier was up to. Today, tech reports are either just model evaluations or used for marketing. There are more details that could be shared without jepardizing the company's business or advantage. Again, a lot of the details are anyway known between the labs. Start with the methods, evaluation setup and failure cases.
Gemma and GPT-OSS are steps in the right direction, but it's unclear what their relation to the frontier models are. We need more such releases and more transparency around them.
Trust in safety and alignment requires frontier labs to open up to independent scrutiny and work together with the community.
mrinal retweeted
What does it mean to pace the frontier?
Over the last month, @random_walker and I have analyzed the loss-of-control incidents at AI companies to understand what technical and policy interventions can improve safety and what companies should do to pace the frontier. The result is a new 13,000 word essay — our most substantial writing on AI safety since AI as Normal Technology.
A summary of our arguments:
1) The polarization between the cybersecurity and AI safety communities is counterproductive. The safety community largely sees these incidents as a crisis for alignment, and worries that these incidents will become more damaging as agents become more capable. Cybersecurity practitioners largely see companies failing to take basic security precautions. We offer a middle ground between these communities as a way forward for improving AI safety.
2) We agree with security practitioners that OpenAI did not take adequate protections for controlling their agents. But this is not just a matter of applying 30-year-old security methods to a new domain. Security for AI agents — AI control — while important, is not a solved problem. While known control methods would have prevented the Hugging Face incident, as agent capabilities continue to advance, we will only be able to control them if we invest adequately in control interventions.
3) We also agree with security practitioners’ implicit position that these incidents are primarily a security story. In the AI safety community, rogue agents are treated as inherently catastrophic because of the assumption that there is an endless list of risks that will arise from their development. We disagree. We have long advocated that the best approach to AI safety is to identify the risks and address those specific risks. Over the last few months, it has become clear that one urgent risk is cyberoffense, because it has unique properties that allow agents to carry it out autonomously. We should similarly invest in defenses against other specific risks, such as biorisk and risks from military AI.
4) We agree with the safety community that there is an urgent need for technical and policy interventions to prevent loss-of-control incidents. But in our view, marginal investments in control are more likely to be effective compared to those in alignment. We view these incidents as illustrating the lack of emphasis on AI control within companies, despite the availability of known techniques. More broadly, there are many common-sense policy proposals that could help promote investments in AI control where we share common ground with the safety community.
5) Organization governance should be a key tool for pacing the frontier. Unfortunately, AI companies are trying to reinvent basic aspects of organizational governance as a problem to be solved by improving the technology. But even developing better control techniques will not be enough if irresponsible individuals or teams within large organizations can choose not to use them. When a single misconfigured RL environment or unmonitored evaluation can cause real-world harm, individual teams should not be able to run potentially dangerous experiments without oversight from legal, security, and other teams. AI companies need processes for reviewing experiments, assigning responsibility for monitoring them, and investigating warning signs deeply before restarting experiments. If putting these processes in place requires pausing some experiments, companies should do so.
6) How should we reason about AI's impact on cybersecurity? It's plausible that advances in agent capabilities upset the offense-defense balance for cybersecurity. We cannot yet be certain, but there is enough evidence that agent capabilities might soon make widespread cyberoffense possible that urgent action is warranted. We discuss potential interventions for tilting the offense-defense balance towards defenders.
7) How our views have evolved over the last year. We take stock of AI progress and share how we have updated our views. In the essay, we did not pay sufficient attention to safety risks that arise during development and evaluation (as opposed to the widespread deployment of models). We were too confident that companies would take basic control precautions and underplayed the importance of jaggedness, which led us to underestimate how quickly capabilities could improve in domains such as cybersecurity.
8) At the same time, many distinctive claims of AI as Normal Technology have held up. In particular, we think recent incidents support our continuity hypothesis — the behavior of "rogue" agents became apparent and widely publicized while they are still incompetent at causing serious harm or hiding their traces. The societal reaction to even the relatively small harms from these incidents has been fierce (and the safety community deserves credit for keeping up pressure on companies). Whether this translates into meaningful changes in companies’ behavior remains an open question, and a test of the usefulness of the AINT framework.
9) In short, we’ve tried to synthesize the AI safety and cybersecurity communities' views into a coherent plan of action: hold companies responsible, invest in control, and strengthen defenses against specific risks.
ALT table with the summary our our arguments
mrinal retweeted
Sure, but when it turns out the leak was caused by the plumber who neglected to tighten a pipe fitting, or perhaps poked a hole in one of the pipes, you shouldn't question the entire plumbing industry. You should question the plumber's compétence and/or motivations.
mrinal retweeted
I've been trying to think of what to say (if anything) about the recent discourse around Navier-Stokes, AI math, etc.
I then realized that one of my heroes, William Thurston, already said everything that needs to be said—sixteen years ago:
mathoverflow.net/a/44213
