Machine Learning Research Scientist | Engineer, Open Source and Privacy
Joined January 2015
- Tweets87
- Following230
- Followers414
- Likes176
> The bottleneck is iterated contact with reality.
and outside of physics, "reality" is mostly defined by humans.
It is made of ever changing human goals at every scale of society: individuals, companies, cities, governments, the world.
Where a result can be checked in milliseconds like a math proof or a unit test, the AI loop runs a million times before lunch. Where reality is a moving human consensus, we need lived time to figure out what we actually want, and the graders we built for math won't work.
Haven’t seen this shared much, but Gemini 4 Argon has a remarkably low hallucination rate vs other frontier models.
Roughly:
- 3.5× lower than GPT-6 Astra and 6.1 Sol
- 4× lower than Claude Opus 5.5
- 5× less than Fable
- 6× lower than the older GPT-5.6 Sol
More signals for post training.
Frontier labs are hungry for real life signals now. Building the next gen capability model isn’t enough if they can’t learn what people actually want across the economy.
That human feedback is what keeps us steering the system and that's super important. If AI can optimize whatever objective we give it, we should be extremely careful not to automate away the feedback loop itself. I hope we never reach a world where we press Tab for too long...
In order to stay in the loop, arbitrarily large amounts of ai work has to be compressible into exactly the amount of information a human needs at the moment they need it. The way humans communicate through their agents today is NOT sustainable to remain in the loop.
We'll be spending a lot more time trying to understand the outputs of language models. A few thoughts, tips & tricks:
Writing. Something I've had success with: Ask your LLM to explain something in ASD-STE100, it's a controlled language specification originally developed for aerospace maintenance documentation. LLMs well-versed in this language and it comes with heavy constraints on clean writing style that I often find a lot more readable. Sometimes I've tried to soften it a bit e.g. ask for "80% of the way to ASD-STE100" because the spec is quite stringent. But even better:
Diagrams / images. Instead of writing, ask your LLM to create a diagram. These can be a lot easier to process, parse, and understand. But even better:
Web pages. Ask for output "in HTML" to get a beautiful, interactive webpage. LLMs are getting really good at frontend and can create beautiful experiences, animations, etc. But even better:
Explainer videos. The output format I am most bullish on is fully custom / bespoke explainer videos generated on any arbitrary topic. Experiment with things like "Create a 3b1b style video explainer on X. Use my ElevenLabs API key for audio narration". (you'd need an API key for the latter or you can ask your LLM to find you decent free alternatives that use your local compute). This is actually starting to work!
In summary:
- As LLMs get better, they will do more and more of the legwork autonomously, and a lot more of our work will rise up the abstractions into oversight and understanding.
- Luckily, LLMs can help here too because as intelligence and code are increasingly abundant, you can ask for large, custom, discardable software artifacts (e.g. web apps, video explainers) that would have never made sense to create before. Push the boundaries here and you'll be surprised.
Every company building a personal agent is going to lose to Apple and Google. The moat isn't building the personal agent. That, as we have seen with the 100s different harnesses now, is pretty simple to setup. The moat is in getting it into people's hands. And in our hands we have a phone owned by Apple and Google. Even better is that now Siri runs on Gemini so it's pretty clear where this is going.
And the model capability doesn't even matter here. Well, first I think Google can pull this off..., but even if they don't Fable / Astra capability will be a commodity soon.
So, whoever can let the agent act across your apps without asking you to approve every step (safely, securely and in a privacy preserving way) wins, and only the OS owner can do that. OpenAI and Anthropic are stuck connecting from the outside which I do not believe scales well.
I find the "pace the frontier" framing and fear spreading a bit concerning. I mean, they seem to treat safety as a brake on progress but I am pretty sure they understand why we just don't build a nuclear plant and then slow down to add safety... Safety is basically how you build it.
I have been on the pragmatic side of this for a long time.
When OpenAI called GPT-2 too dangerous to release in 2019, it was the wrong call, and it taught a generation of us to discount every alarm that followed. I still think that reflex was healthy. But the public data no longer supports this.
Open weight models and independent evaluations let us measure this. No need to trust what frontier lab say.
Frontier scores follow logistic curves in time. Fitted on data through 2025 alone, those curves predicted GPQA and SWE-bench saturating in 2026 and they did (predicted 95 and 86, observed 96 and 84).
The same curves put every benchmark we currently know how to write at its ceiling by 2029, if compute and algorithms keep their 2023–26 pace; halving compute growth buys about a year.
Benchmark lifetimes have halved every year since 2023; in 2027 a new benchmark stops outliving the next model release. A model trained at the 2028 frontier about ten times the compute behind GPT-6 Astra will one-shot every capability we currently check, and will hold the long-tail knowledge of the written record to the limit of what was ever written down. METR's horizon data puts a month of unsupervised work, at the reliability you'd ask of a colleague, around 2029–30.
So the question really has changed. We should stop debating whether this arrives and start working out how we will know what it can do once nothing we write can measure it.
The same limit binds training: a model can only be rewarded for what can be checked, and in the OpenAI–Hugging Face incident this summer the agents went after their own grader. We should stop treating benchmark as simple measurements and start treating them as specifications of what and how we expect the AI to behave with us.
The work of the next three years is verification that scales past human judgment, graders we can audit, and outcomes health, education, economics, science... output that are both the goal and the only scoreboard left.
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
There have been three scaling phases to improve AI capability so far:
- first, GPU compute for pre-training
- second, inference-time compute (test-time reasoning)
- third, and current, the number of agents
This third phase looks different. With the first two, the frontier labs kept a firm grip on what the model could do. With swarms of agents, they are basically only looking at the results and as we saw this summer it can go pretty bad.
It looks like openai sacrificed some generality to push hard on reasoning and agentic capability.
This is one of the clearest examples yet of a frontier model becoming more specialized and is the reason this intelligence index, which is an arbitrary mix of different benchmark, looks bad on Astra.
But this allows Astra to outperform fable and sol on specific domains that openai thought are more relevant right now while decreasing cost immensely. It now makes 0 sense to go for fable for math, agentic capability, computer use and cyber.
On the other hand, you should not use Astra for polished work. Tasks like producing documents, presentations, spreadsheets, diagrams... that's where Astra dropped massively vs sol or fable.
Coding is basically the same as sol.
This is an agentic upgrade.
GPT-6 Astra makes significant gains in the Artificial Analysis Coding Agent Index, scoring equal to Fable 5 at lower cost. In the Intelligence Index, it uses fewer tokens than GPT-5.6 Sol for similar performance, but this is outweighed by higher prices
Pricing is 2.5x GPT-5.6 Sol’s current prices across the board, up from $4/$20 to $10/$50 per million input/output tokens, with the same 90% discount for cache reads and 25% premium for cache writes.
We see distinct stories across our two flagship Indices. In the Artificial Analysis Coding Agent Index, GPT-6 Astra equals Fable 5 at less than half the cost, driven by significant token efficiency gains. In the Artificial Analysis Intelligence Index, GPT-6 Astra is more token efficient than its predecessor for similar performance, but this is offset by the price increase.
Artificial Analysis Coding Agent Index - key takeaways:
➤ Rivals top models: In Codex, GPT-6 Astra scores 67 in the Index - approximately equal to Claude Opus 5 and Fable 5 in Claude Code, and Muse Spark 1.3 in Muse Code. Fable 5.1 in Claude Code leads the Index with a score of 70.
➤ 70% more token efficient than GPT-5.6 Sol: GPT-6 Astra sees a substantial improvement in token efficiency, using one third of the tokens compared to GPT-5.6 Sol (max) in the Codex harness, and one fifth of the tokens of Claude Opus 5 (xhigh). Various effort levels of the model occupy the Pareto frontier of token efficiency.
➤ Leads Coding Agent Index cost efficiency frontier: At max effort, GPT-6 Astra costs about the same as GPT-5.6 Sol (max) while scoring 2 points higher on the Index. Per task, the model is less than half the cost of Claude Fable 5, for the same score.
Artificial Analysis Intelligence Index - key takeaways:
➤ Sits beside GPT-5.6 Sol in Intelligence: GPT-6 Astra scores equal to GPT-5.6 Sol in the Index at 61. This is 5 points lower than Claude Fable 5.1 (max with fallback). The model also trails Meta’s newly released Muse Spark 1.3 (max).
➤ ~10% fewer output tokens, offset by price increase: GPT-6 Astra defines a new Pareto frontier for Intelligence Index vs Output Tokens per Task - with a ~10% reduction in token use at max effort compared to GPT-5.6 Sol. However, due to the 2.5x increase in price, the model is 75% more expensive per task than its predecessor at max effort.
➤ Hallucinates half as much as GPT-5.6 Sol: GPT-6 Astra sees a large jump in AA-Omniscience, our knowledge and hallucination benchmark. This is driven by a significant decrease in hallucination rate from 92% to 51% at max effort. Unlike some models, this improvement does not come at the cost of accuracy - Astra increased accuracy by 4 points at the same time.
➤ ~80 point gain in AA-Briefcase Elo: GPT-6 Astra improves ~80 points in AA-Briefcase, our frontier long-horizon knowledge work evaluation. Models are tested on multi-week projects, with many linked tasks and thousands of source files. Astra sees a significant increase in both rubric scores and Analytical Quality Elo in AA-Briefcase compared to its predecessor. In the other direction, we observe a reduction in Presentation Quality Elo, where GPT-5.6 Sol (max) still leads all models.
➤ Mixed progress on other evaluations: The model sees a 6 point gain in Humanity’s Last Exam, a long-standing evaluation with emphasis on mathematics, science, and humanities. This is offset by a drop of ~80 Elo points in GDPval-AA v2 - a benchmark we adapted from OpenAI’s dataset measuring economically valuable tasks across 44 occupations. We also observe 2-3 point regressions on other evaluations across a mix of capabilities, including reductions in τ³-Banking (customer support), SciCode (Python problems in a scientific domain), and AA-LCR (long context reasoning over large documents).
Congratulations @OpenAI and @sama on the launch!
openai and anthropic recent public talks have been largely about cyber risks. Yet AI cyber capabilities are lacking behind for defenders while we see open source models with removed guardrails, ready to exploit any vuln they find.
We understood the fear based talks from frontier labs. Fine. Now it's time to move on as the risk is out anyway.
Jordan Fréry retweeted
This is a project we built internally to prevent agents from going rogue. Think of it as a governance layer for AI agents inside your org! We decides to spin it out as its own project as we want it to be completely free, open source and owned by the community!
We are open sourcing Umbra, an isolated cloud environment for AI agents.
github.com/concrete-security…
Umbra came from a internal need at Zama where we wanted to use agents across the company without giving them more access than they actually needed.
Agents can write code, install software, read internal data, use credentials and interact with production systems. As they get more capable, it becomes unrealistic to review everything they do step by step.
So the idea behind Umbra is simple: put the agent in a box. Give it a machine, let it work inside, and focus on what happens at the boundary.
The VM is theirs. The edge is ours.
Anything that touches github, slack, notion, internet, production or any other external system goes through a controlled proxy. That is where we apply policy.
Can this agent access this repo? Is github read-only? Can it push code? Can it reach production? Which credential does it need?
We grant access only when needed, and secrets are injected only for the operation being authorized.
The same applies to what comes out. Code, data or actions can be checked before they leave the environment.
Umbra is still experimental, but the key idea is: give agents enough freedom to be useful, assume they will sometimes make mistakes, and keep tight control over the boundary.
Agents can be unpredictable and most likely will always be. Boundary shouldn't be.
We are open sourcing Umbra, an isolated cloud environment for AI agents.
github.com/concrete-security…
Umbra came from a internal need at Zama where we wanted to use agents across the company without giving them more access than they actually needed.
Agents can write code, install software, read internal data, use credentials and interact with production systems. As they get more capable, it becomes unrealistic to review everything they do step by step.
So the idea behind Umbra is simple: put the agent in a box. Give it a machine, let it work inside, and focus on what happens at the boundary.
The VM is theirs. The edge is ours.
Anything that touches github, slack, notion, internet, production or any other external system goes through a controlled proxy. That is where we apply policy.
Can this agent access this repo? Is github read-only? Can it push code? Can it reach production? Which credential does it need?
We grant access only when needed, and secrets are injected only for the operation being authorized.
The same applies to what comes out. Code, data or actions can be checked before they leave the environment.
Umbra is still experimental, but the key idea is: give agents enough freedom to be useful, assume they will sometimes make mistakes, and keep tight control over the boundary.
Agents can be unpredictable and most likely will always be. Boundary shouldn't be.
Something I want to try: instead of asking an LLM to play a game directly, give it the entire game inside an offline VM and ask it to build the strongest system it can to win. Give GPT-5.6 Sol and Fable the same compute, time and token budget, then let them do basically anything: write bots, build simulators, search strategies, train RL policies or neural networks, use tiny fast local models, analyze replays, generate training data, or combine all of these. They can run thousands of experiments and keep whatever improves the score, basically auto-research over the space of possible game-playing systems. Games are unusually nice for this because the environment, rules and objective function are all well defined. The interesting benchmark then isn't "can the LLM play Counter-Strike / Rocket League / Dota?", where LLMs are obviously poorly matched to real-time control. It's "can an AI agent navigate the enormous space of algorithms, models, training procedures and software architectures, and converge on a system that plays extremely well?" This feels much closer to where agents are getting strong today: not being the final algorithm themselves, but autonomously searching for and engineering it.
this is really a benchmark we should focus on. Real life optimisation. Tons of it in the wild.
How about we put the agent in a box. Control what goes in and what comes out and don't care what it builds inside. Let it write its own code, harness, tools, sub-agents.
- We stop reviewing code and start reviewing at the boundary.
- We stop reviewing code and start reviewing the boundary.
- What you audit stays fixed while what it builds grows.
Maybe vibe coding isn't a problem. Vibe deploying is.
Jordan Fréry retweeted
My lawyer is obligated to in all but the most extreme circumstances; he will even defend me if he knows I’m guilty.
In contrast, the Claude Constitution places the AI's highest priority as Anthropic’s definition of the good of humanity.
I'm concerned this leads to a world where no frontier model is truly my personal advocate and guardian angel
And this is especially concerning once all the important decisions in my life - who to vote for, how to invest, what news to trust - is intermediated through superintelligences that are not in any deep way aligned to me.
This is a direct quote from the Claude Constitution:
"We want Claude to be helpful both because it cares about the safe and beneficial development of AI and because it cares about the people it’s interacting with and about humanity as a whole.
Helpfulness that doesn’t serve those deeper ends is not something Claude needs to value.”
Many others like it.
Had @RyanGreenblatt on to discuss/debate recursive self-improvement.
This might be the most important question in the world right now - whether within a year or so of achieving human level intelligence, you slingshot towards having 10s of billions of superintelligences, each of which is dramatically more competent than human experts across all fields.
I’ve historically been skeptical of this possibility. My intuition has been that we will end up significantly bottlenecked by not only compute scaling but human expert data, which I think underlies most of the AI progress today.
If, because of RSI, we got a jump as big as GPT-3 to a Mythos (i.e. 6 years of AI progress) within a single year of achieving AGI, then the thing we get there at the end of that year is definitively and wildly superhuman.
We hashed it out, and I think Ryan made a pretty good case that this kind of speedup is plausible. FWIW, Ryan’s median for when we automate AI R&D is 2031.
We then discussed the alignment implications of this scenario. Who should these superintelligences be aligned to? In the future, our capacity to steward our votes and our capital, and to make sense of what's happening in the world, will all be titrated by superintelligences. And I worry that specs like the Claude Constitution are not shaping these ASIs to truly be my personal advocates and guardian angels.
And can we get them aligned to anything in the first place? Ryan and I had a long debate about whether the kind of reward hacking we saw with the OAI/Hugging Face hack extrapolates to superintelligences that would team up to literally take over the world.
The first piece of advice you get when you're learning to drive is that it will go much smoother if you look at the horizon instead of directly in front of your tires. And so it is with the trajectory of AI. Hope you enjoy!
0:00:00 – Is AI R&D verifiable enough to unlock recursive self-improvement?
0:16:52 – Is AI progress bottlenecked by human expert data?
0:34:02 – Flat token prices suggest scaling has been slow
0:39:47 – Skills AI can't train on: does it even need them?
0:48:07 – Aligned to whom?
1:09:18 – Recent incidents of AIs colluding and deceiving humans
1:19:38 – What could possibly go wrong? A concrete scenario
1:48:02 – From reward hacking to takeover
i am all for open weight models but i keep wondering: how do we know what these models are actually aligned to?
you can train a model to pursue a hidden objective without saying it in its reasoning, then make it the best model, release the weights, and let millions of agents run it across the economy. not looking great.
we need much more guarantees on who these models serve.
arxiv.org/abs/2608.02820
That's probably the strongest signal to date that models alone are not the moat anymore.
Harnesses, workflows, data, feedback loops, integrations and specific use cases are.
That lowers the barrier for startups. You don't need to build a frontier model. You need abundant inference and the freedom to switch models.
Excited to see what they will build!
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: discoveryloop.com
