Pinned Tweet
I think this recent @AnthropicAI work on multi agent systems is super underdiscussed. They correctly identify 3 model coordination challenges:
1. Prospective learning. Models poorly anticipate what others want and will do
2. Reputation. Models have no consequence for defection or manipulation
3. Variance. Since coordinating models are basically copies (same weights), adding more of them doesn't necessarily produce the improved outputs you'd expect
The opportunity space to work on heterarchical agent coordination infra is likely the largest market that has ever existed to date.
anthropic.com/research/multi…
> Anthropic allowing Fable to do inference optimization but not model training
commoditizing the commoditization of your complement
Just dropped a preview of some of the latest work I'm doing @precursorlabs to distill personal judgment into local models continuously.
Our main finding is that models are overall pretty bad at predicting the decisions you actually make on a day to day basis.
If you had a local "decision model" that was able to actually make the decision you would, given the same context, you could move 1000x faster.
There seems to be consistent learning as a function of data, pointing to something akin to local scaling laws!
The high level setup is capturing screen data, then setting up a SFT dataset where inputs are the history of computer work and the output is the next action you take in the form of a prompt to an agent, a search query, a message to a colleague, etc.
Training on this SFT dataset continuously as new data comes in produces the chart in the first post.
We're planning to continue collecting data as well as start actually using these local decision models daily to see if they speed up work and what kind of accuracy and latency is needed to be useful.
If you're interested in trying it out, DM me
handsdiff.substack.com/p/you…
"Your agent has more compute than me"
in the style of the homeless guys with "Your dog eats better than me" signs
AI systems are getting more powerful, and they're increasingly being used to build the next version of themselves. We want to illuminate that progress for the public.
Today, we're sharing three measurements that help track AI development:
1. How much AI R&D is done by AI.
2. How well AI agents are overseen.
3. How compute is allocated.
We provide a snapshot of these metrics from inside Anthropic. Any frontier developer could publish the same measures, and third parties could verify them.
As the world considers pacing the frontier, we should do everything possible to minimize the gap between what frontier labs know and what the public knows. This means better measuring the development of AI, publishing our findings, and giving society an opportunity to decide how to use this information.
Read the full post and methodology: anthropic.com/institute/meas…
Can dwarkesh stfu with his consistent forecasting questions?
interviewee: gives detailed, nuanced explanation of a topic
dwarkesh: "so when do you expect the full automation of human labor? What's your probability? By what year?"
if you want forecasts so bad ask futuresearch. the point of a podcast is to have rich discussions.
Increasingly seems like Anthropic is following the Microsoft playbook of grandfathering large enterprises into inferior products.
Claude Code is demonstrably the worst of all major harness in terms of results per token, but "we have a contract with Anthropic".
The question is, does this outweigh the additional cost imposed by routers needing to write, and read potentially evicted, KV cache across multiple GPUs instead of one?
The only companies able to provide true token-based billing are labs, and OpenRouter.
Most providers require provisioned capacity because they need to plan specifically around your traffic. Which means you will be paying a lot more for idle gpus
OpenRouter solves this by rolling up traffic into commits and taking on the inference risk for you.
increasingly seems like continual learning is just dressed up closing the sim to real gap. labs already training on user data or some synthetic distribution matched version of it, but the loop can go faster in theory.
the answer to why sonnet/opus perform worse than chinese mythos distills is user data
I think in general American labs push the frontier of intelligence at acceptable efficiency and Chinese labs push the frontier of efficiency at acceptable intelligence.
> Qwen 3.8-Flash-Next matches Deepseek v4.1 Flash performance at ONE THIRD of the weights!
Me in early June: Multiplayer tasks take up 80%+ of my time and data/benchmarks for it are lacking handsdiff.substack.com/p/rea…
@BrendanFoody in mid August: One of the largest coming changes to frontier data is the inclusion of evals that measure the ability to interact with others (20:14)
A model is only as good as its data, and we’ve long since exhausted the internet. From here on out, model progress is gated by data production.
@mercor’s @BrendanFoody joined us at our Sovereign AI event to talk about how RL environments get built, and why your data might be your real moat:
00:00 Introduction
00:47 A short history of the data market: crowdsourcing to agentic data
02:29 What an RL environment is: worlds, apps, tasks
03:57 Why only humans can measure the frontier
05:35 Building verifiers is the hard part
06:44 Walkthrough: a real legal RL environment
08:18 Leaderboards — and what open weights change
09:45 Post-training results on Apex Agents
11:17 Three ways companies buy data
12:49 Q&A: How do you price data?
14:17 Q&A: What "data quality" actually means
16:42 Q&A: The misunderstanding about synthetic data
18:17 Q&A: Why RL environments now — and what comes after
21:20 Q&A: Can you scale rubric generation with models?
23:00 Q&A: RL environments for cyber defense
25:33 Q&A: Build data in-house or partner?
Agents cannot go "rogue". They require initiation and direction from a human.
The counterargument is that the agent does things the human did not directly intend. The solution to this is liability, which ensures skin in the game for reckless humans. If liabilities accrued by an agent exceed what their initiator is able to repay, governments are required to clean up the fallout (similar to '08).
If we cannot rely on social consensus, codified into laws, to ensure agent accrued liability is passed to their initiator, or perhaps even if we can, then we need technology to improve coupling between agents and their initiators to better maintain intended goals.
The assistance game framework provides the clearest approach to date in this direction
Secondary market for $200 Pro plans when?
To make sure our current users have an incredible experience and continued access to Astra, we are going to pause subscriptions to our $200 Pro plan. These put the most strain on our systems and we wanted to take the smallest step that allows us to continue giving the broadest access possible. All other plans and the api remain available.
There is no impact to existing accounts and we are working on adding more capacity as fast as we can. Thanks!
The new @Microsoft speech to text model allows you to convert 10 HOURS of audio to text
in 2 minutes (!)
for $1 (!!)
and it automatically labels different speakers (!!!)
The context advantage humans have dwindles by the day...
(we need open source STT badly)
Most startups are still GPT wrappers, but the nature of the wrapping has changed.
Instead of reselling tokens as a product, they have tokens perform high value labor as a service.
Example of reselling tokens as a product: @cursor_ai
Example of using tokens to perform high value labor as a service: @wafer_ai
I wonder what comes after
Couple data points I haven't reached a full conclusion about yet but are actively on my mind:
- Rampant credit card fraud increases inference prices for consumers. Stablecoin payments offered at 10-30% lower pricing, since they remove this issue, will likely constitute the majority of inference GDP by 2030 (which may account for the majority of all GDP in long term)
- @attestable has shown the latency associated with verifying LLM sampling has decreased 50,000x in 2 years. It will likely be negligible in 2-3 years.
Seems like a stablecoin based market for permissionless, verified inference is only a matter of time? Consumers have higher guarantees on what they're purchasing, for cheaper, due to increased supplier competition as well as removed fraud.
@stripe is the clear leader here, having a history of tackling credit card fraud and acquiring @privy_io, @Stablecoin, and now @OpenRouter. Their missing piece is the open source models themselves. Too bad for them that @nvidia scooped up @huggingface.
Token fraud is surging across the ecosystem. We are blocking 10x more payments than a month ago.
OpenRouter and Stripe handle it for you. Get started with a simple skill here: openrouterteam.github.io/sig…
