Building a living world model of the investable universe at https://xitter.cf/t.co/od3lfCDlbw
Cascais, Portugal
Joined July 2021
- Tweets1.4K
- Following868
- Followers619
- Likes5.9K
Pinned Tweet
We analyzed the Anthropic IPO using our probabilistic causal model of the economy. It predicts a brutal collapse in the medium term, with less than a 10% chance of holding its IPO price a year out, and only a 0.2% probability of a long-term 10x return. (OpenAI's tells a similar story.)
When we decompose the gap between market price and intrinsic cash-flow value across the AI stack, the incumbents are held up by an S&P 500 passive index floor, while 95% of OpenAI's and Anthropic's valuation is unanchored private-mark expectation. Floating a small sliver of shares can manufacture a day-one squeeze, but once lockups expire, supply floods into into a market with no passive index bid, ongoing dilutive issuance, and over $600B each in take-or-pay spending commitments.
There are scenarios where this ends well: where margins and top-line growth stabilize at levels high enough to clear the commitments, and both companies become incumbent cash cows. But given today’s landscape, it’s 9 times more likely that margins will continue being competed down, while capability competition will push for ever-higher capex. Meanwhile, Anthropic’s Q3 already shows signs of demand losing steam.
Early private investors in Anthropic and OpenAI may yet earn massive cash-outs. Our model indicates that IPO entrants will be taking on startup-like risks without the startup-like returns.
Imas et al. argue that AI only transforms the economy once agents learn to coordinate. They describe a "supercooperative threshold": past it, AI-only firms grow without limit and the economy concentrates.
I agree with them in that the individual agent is the wrong unit, and that what matters is how output scales with organization. But their model tracks only two state variables: competence and coordination. It leaves out the one that actually governs how problem-solving scales: the **world model** (WM). That is the stock of verified, reusable structure a system accumulates instead of re-deriving solutions.
What firms coordinate around is their WM. Firms coordinate by sharing routines, standards, specifications and models. Coordination is cheap where these exist and expensive where every handoff has to rebuild shared context from scratch. The railroad and telegraph, their own example, made handoffs reliable by crystallizing infrastructure. They didn't make workers better communicators. In our framing, coordination cost is mostly the cost of search, and reusing verified structure is what removes it.
This is a simpler explanation for the convex returns they attribute to a "supercooperative threshold" (and one which follows economic history rather than predicting a break from it). Re-deriving a high-stakes solution costs exponentially more as the task gets more complex, because every constraint must hold at once. Reusing a verified one costs a bounded grounding step. A WM is therefore a **capital asset**, which gives increasing returns at every capability level and at any compute price. This pull toward accumulation needs no coordination ability and no foresight. It also gets stronger as capability rises.
Agent copies sharing weights is **not** what we're talking about. Shared weights give you copies of the same reflexes. They don't give you warrant: evidence, independent of the generator, that a solution satisfies the task's constraints at deployment stakes. Warrant comes only from accumulated track record. Generation can be scaled; warrant can only be accumulated. So the economic transformation is gated by **how fast verified WM coverage builds up against drift in the world**, not by a capability threshold that agents cross.
That changes the picture of the transition. Coverage is high in deep, recurring, slowly drifting industries/markets, and low where tasks are novel or drift fast. In those low-coverage industries, search, and therefore humans, legitimately persist. That explains their niche-market finding instead of just assuming it.
And it changes the policy question: Whether the economy concentrates depends on who owns the WM. One that's governed as a common good spreads the increasing returns across principals (as open-source software has shown for decades). Proprietary WMs locked inside a few AI-only firms are bound to produce the concentration that Imas et al. predict.
I joined @GoogleDeepMind to develop a research program on the economics of AGI. Today, together with Andrew Koh (@andrewjkoh) and Shane Legg (@ShaneLegg), we are incredibly excited to release The Economics of Intelligence, the first in a series of projects from our group on how AGI will change the economy.
How do capability improvements transform the economy, and when? In this project, we define thresholds of economic capabilities, e.g., minimal AGI, based on how they transform the economy. We then track how the economy–market structure, employment, and wages–changes during the transition across capability regimes.
We have seen exponential increases in what AI agents can do but the broader economy seems almost remarkably unchanged. Part of the reason is that the solo AI agent is the wrong economic object; economic transformation is an equilibrium outcome where organizations compete. Looking at how intelligence functions within organizations, transformation will depend on when two key capability thresholds are crossed: competence (gains on individual skills, e.g., coding, math) and coordination (gains on ability of agents to cooperate and work together).
We show that what economic transformation looks like depends on whether AI becomes supercompetent, supercooperative, or both—the regime of artificial superintelligence (ASI). Tracking this transition shows why the real visible transformation may come from improvements in coordination—a capability most benchmarks don’t yet track. Past a certain threshold, AI capabilities can break the Coasian ceiling on firm size and lead to an economy that looks drastically different than the one we see today.
Two main takeaways:
1. To understand the path of transformation, we should be tracking agentic coordination more closely.
2. The impact of AI will likely show up in wages before market structure or employment.
We are really excited about this paper. Excited to hear people’s thoughts.
Link: institute.deepmind.com/essay…
Rafael Kaufmann (🧠,🌎) retweeted
First issue of Primordia Intelligence is out.
Three ways the AI story could break. Which companies absorb it and which ones don’t.
Guess before you read, then check your work.
primordiaintel.substack.com/…
Active Inference is optimal control! This hunch has been with me since 2024; now it's a theorem.
* In a Markov-blanket dynamical system, the Expected Free Energy (EFE) of a policy is the relative entropy of the path measure obtained under the policy vs its preferred path measure.
* I prove that this functional is equal to the expected cost of the policy in the usual MPC sense: cost of control (active + cognitive) + terminal value.
* So, in continuous form, minimizing EFE is path-integral control. Its canonical discretization is KL control.
* The exploration-exploitation tradeoff celebrated by ActInf is just the classical dual control effect: exploration emerges from closed-loop control under partial observability, without any epistemic/information-gain bonus.
* The two EFE formulations in the literature (epistemic loss + pragmatic loss and risk + ambiguity) suffer from known pathologies; I show our formulation avoids those pathologies.
* In simulations using `pymdp`, the canonical EFE weakly dominates both previous formulations, performing optimally under settings where those struggle. I implement it both using tree search and efficient Bellman dynamic programming.
* As a bonus, "G-theory" (ActInf's extension to systems without steady states) is resolved as path-integral control on the jet bundle.
So what? Here are my 3 big payoffs:
1/ ActInf enthusiasts don't have to defend their loss functional of choice; it is uniquely determined by model and preferences, and is identical to standard practice in control theory. They can also borrow highly efficient computational methods from both continuous and discrete control.
2/ Control theorists and engineers can now price the costs of action and cognition on equal informational footing.
3/ Utility maximization, Herrnstein probability-matching and habitual behavior turn out to be points in a continuum parameterized by cognitive capacity.
I hope that, by strengthening the bridge between ActInf and control theory, both fields will benefit. Code is available upon request; for feedback and discussions, DMs are open, but email is best: rafael.kaufmann@gmail.com
papers.ssrn.com/sol3/papers.…
Intriguingly, these two systems may be complementary in ways that reduce some of the huge efficiency tax inherent to LLM-based workflows (see our paper research.primordia.ai/gwm-v1…). But that's yet to be seen!
Jev (System 1): A general classifier // An LLM without the "L"
* Efficiency without explanation
* "Decision quality" means "trust us, we've calibrated it"
* Output structure/type-safety as syntactic constraint
* Will gladly make confidently wrong decisions about anything
GWMs (System 2): Modeling the world
* Efficiency with full explanation
* Decision quality means "Bayes-optimal decisions given existing knowledge"
* Output structure/type-safety follows from model structure
* Will only make decisions about what it knows
Full intellectual honesty: I 100% got carried along by the hype. After testing Jev and a few of the OSS alternatives, I can confidently say: This is *not* "the same overall architecture" as our GWMs, and the benefits are almost diametrically opposite:
This is a world model of structured, observable systems. Same overall architecture as our GWMs, pointed at a different kind of problem but bringing the same benefits: orders-of-magnitude improvements in efficiency and decision quality. Fantastic.
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
Rafael Kaufmann (🧠,🌎) retweeted
BREAKING: OpenAI might have stolen another major proof.
In a detailed Mastodon post, which I report in full in the comments, Andreas Thom presents several pieces of evidence suggesting that OpenAI may have trained Astra on conversations in which he and Gábor Kun were working on Gromov’s soficity conjecture, one of the ten problems OpenAI later announced Astra had solved.
I know Andreas. We met several times early in our careers. He is an exceptional mathematician, a leading expert on sofic and hyperlinear groups, and one of the most respected scholars in the field. He has spent two decades working on this problem.
If his account is correct, this is not a minor dispute over attribution. It would mean that unpublished human work was absorbed into a model and then presented to the world as a breakthrough by the model itself.
And if the allegations raised by Levent Alpöge, Tristan Buckmaster, and now Andreas Thom are all substantiated, we are no longer looking at isolated incidents.
We may be looking at one of the greatest intellectual scandals in the history of science.
AI is not discovering new mathematics.
AI is stealing human discovery.
Rafael Kaufmann (🧠,🌎) retweeted
go fuck yourself @sama
claiming that you solved navier stokes because 10000 agents ran in circles for 88hours on a multimillion dollar gpu cluster to formalize in lean a blowup case under controlled external forcing is pure scientific vulgarity the clay mathematics institute millennium prize does not ask m whether you can artificially force a singularity in a fluid by injecting an ad hoc smooth external forcing term f(x,t) to twist the vortex until it breaks the real problem questions the fundamental stability and global smooth existence for 3dimensional incompressible euler & navier stokes equations under natural conservation laws and viscous dissipation alone using a mathematical loophole on forced equations to parade a century old victory is a major conceptual scam Altman
technically & epistemologically what you present as an agi breakthrough is nothing more than bruteforce combinatorial autoformalization the ai did not understand fluid mechanics it simply navigated a continuous search space previously mapped out and constrained by the monumental work of human mathematicians like tristan buckmaster/ levent alpöge / diego córdoba or tarek elgindi coordinating 10000 agents to check the logical consistency of a 100 page proof via lean is a software engineering feat and computational parallelization triumph not an intrinsic scientific discovery it is the victory of the compute bulldozer over abstract human intuition repackaged for the public as a higher mathematical consciousness
to this theoretical imposture you add a disgusting ethical and industrial cynicism taking advantage of private codex sessions and informal preprints from academic researchers to siphon their research leads and then trying to redact or erase the contribution of levent alpöge under the pretext that he works at rival anthropic is intellectual serfdom openai behaves like a feudal lord of silicon appropriating the cognitive subsistence of independent scholars threatening their careers behind closed doors if they protest and turning community academic labor into a privatized pressrelease
this entire staged event serves a desperate financial agenda in a pre ipo panic facing the slowdown of scaling laws and growing investor skepticism over the profitability of foundational models openai needs to manufacture an artificial sputnik moment claiming to solve a millennium prize without immediately submitting the proof to traditional peer review means using the prestige of fundamental mathematics as cheap marketing fuel to inflate a delusional valuation!!!
real science is not a clout chase on social media or a compute spike spent to rob the clay mathematics institute it is a quest for elegance physical truth and universal rigor to decode reality true artificial intelligence will not emerge from hostile corporate takeover of academic work hidden behind computational bruteforce but from architectures capable of generating new conceptual paradigms by masquerading constrained formalization as the collapse of physics greatest mysteries you did not solve navier stokes you only proved how far silicon valley will go to prostitute scientific integrity for capitalist spectacle
We’re sharing a solution to the Navier-Stokes Millennium Prize Problem, one of the deepest problems at the frontier of mathematics.
The proof was produced by a group of agents, using an OpenAI next-generation model significantly more capable than GPT-6 Astra.
The problem concerns whether the description of smooth three-dimensional fluid motion modeled by the Navier-Stokes equations can break down. It has remained unresolved for roughly 90 years.
Rafael Kaufmann (🧠,🌎) retweeted
opus 5 is a sign that the obsession with “long-horizon agents” in model training is finally backfiring
i don’t like long-horizon agents, and i’ll explain why they fundamentally don’t work
some people will immediately jump out and say “skill issue”. well, show me one profitable business you built with a long-horizon agent working all by itself - i’d love to learn
so far, the only thing they were able to build that’s even interesting enough for people to talk about are those 3d games that are a partial clone of something that already existed
the reason an agent was able to build a working prototype of complex games like call of duty was that a team of humans already figured out all the requirements years ago for how such games should work, what kind of controls are intuitive, what mechanics are fun etc
all those requirements were already absorbed into the model weights, so when you say “build me call of duty” the model already knows the details. its long horizon execution capability can get all the requirements implemented, which i must say is indeed impressive
but now you can see - the value of long horizon execution has a prerequisite of a massive amount of high quality requirements clearly defined upfront. it took a big team of very talented humans months of effort and many iterations to define that for call of duty
now imagine games like call of duty don’t exist yet, how would we use agents to build it for the first time? we can’t say “build me call of duty” any more. and there’s no way we can define months-worth of game design details upfront
we’ll have to build a tiny prototype of the most basic mechanics, play with it, see if it’s fun, then iterate and expand the complexity. even with the smartest humans, that’s how we work towards something great
we don’t need agents to go dark for a long time, spend tens of thousands of dollars worth of tokens, and come back with a product the agent randomly decided to build - try build something truly novel with this and you’ll see it can’t come up with anything that’s actually profitable (i’ll show you why in a bit)
we need a tight feedback loop where we can collaborate with the agent, plan with it, understand what it’s done, question its approach, apply our judgement, give it real world feedback and iteratively arrive at a good outcome
and that’s exactly what opus 5 absolutely suck at. why? i explained it in more depth with my previous post on how RLVR works - RLVR trains the model to generate code that can pass predefined tests in an isolated environment, which is fundamentally incompatible with the idea of having human in the loop. the more we train the models with RLVR to be “long-horizon”, the less they care about talking to humans
ok now - why do they have to talk to humans? why can’t the models iterate and apply judgement by itself?
maybe one day they could, but not today, due to many limitations. two examples -
1. LLMs today can’t “watch a video” yet. they can look through a lot of screenshots, which is extremely inefficient at observing a high fps animated signal. so anything that requires continuous visual attention is something LLMs can’t do very well
2. LLMs don’t truly understand what’s “intuitive” or “pleasant” for humans. they know what’s already proven to be intuitive and pleasant in the past, but if you present a truly novel concept, it can’t predict whether humans will like it accurately
because of those limitations, human judgment is still needed for almost anything valuable. without humans in the loop, agents will only be able to repeat something that already existed, or go in random directions without true understanding of whether it’s building something useful
in summary, long horizon agents assume requirements all exist upfront. they are fundamentally against human in the loop. and they don’t have true judgement for what humans like
that, my friend, is why they don’t work
Curiously, 4-6 orders of magnitude are exactly what we find in our benchmarks of @PrimordiaAi GWM implementation vs LLMs, in one domain (fundamental investment analysis). As our techniques are general, we believe these gains will scale horizontally.
In the era of base LLM scaling (2022-2024), I believed the LLM line of research would reach a capability plateau (as later seen with base LLMs).
In late 2024, after the o3 test-time compute demo, I changed my views: the new models were showing genuine fluid intelligence, and with this new line of work, the LLM line of research could achieve unbounded capability scaling. "There will be no wall." I talked about it at length on Twitter and in a blog post.
However, looking ahead, I still do not believe that future AI (say, in 15 years) will be based on the LLM stack. I believe it will necessarily have to move closer to its optimal, final form -- symbolic learning. Obviously this is a risky and contrarian belief -- the safe bet would be LRMs. But let's see.
The only meaningful difference is efficiency, not task-specific skill. I believe current techniques are 4-6 orders of magnitude away from optimality in terms of data efficiency and test-time compute efficiency. But far future AI will be near-optimal.