Study work with AI + robots. @MITSloan PhD, @Ucsb Assc Prof, @Stanford fellow, @tedtalks, @skillbenchinc cofounder, CEO Book: https://xitter.cf/t.co/OJQa0by9MT
Santa Barbara, CA
Joined November 2008
- Tweets9K
- Following246
- Followers4.8K
- Likes14.3K
Pinned Tweet
I'm so grateful and exited to share that my book will be published on June 11th!
It's called "The Skill Code: How to Save Human Ability in an Age of Intelligent Machines"
Available for preorder now, here's a bit of the story:
wildworldofwork.org/p/big-in…
Matt Beane retweeted
Making sure I state it clearly:
We can literally patch and prove bug-freeness of the entire software stack running in production. With AI that exists right now for ~$100m.
This means no more software-level hacks. I know it sounds too good to be true, but it isn't. AMA.
We formally verified 123 open-source projects in a partnership with Anthropic. Formal verification at scale is here!
Now we’re launching OSS Verify to scale it to the software we all depend on. We prove it’s crash-free or send you the patches it needs. Free for open-source. Let’s bootstrap the verified software ecosystem: theorem.dev/oss-verify
SO interesting and potentially helpful!
Now time to do this for theory on technology and organizing. Hopefully they have a repo...
Excited to share this new work on mapping economic theory with agentic AI!
The astonishing rise in AI math capability can help us understand formal models in economics.
We do it by using LLM agents to represent theory papers as graphs.
See thread by @andreacicca_ here:
Wondering what humans have left to do in intellectual work?
Read on.
as a founder of @RefineDotInk, discussion around this new paper is v. interesting!
The results confirm that AI reviews are incredible for detailed, actionable feedback to improve a manuscript.
If AI is better, should AI replace human review?
Quick view from the trenches
1/
Matt Beane retweeted
as a founder of @RefineDotInk, discussion around this new paper is v. interesting!
The results confirm that AI reviews are incredible for detailed, actionable feedback to improve a manuscript.
If AI is better, should AI replace human review?
Quick view from the trenches
1/
New paper: "AI Article Reviews Dominate Human Peer Reviews at Their Best: Evidence and Policy Choices" See: GaryKing.org/peerless. with Steven Worthington. Comments welcome!
The jig is up. Research and publishing must change (and are, to be fair - just much too slowly).
A superb account of the key role of occupational identity in prior technological shifts.
Doesn't tread into social theory of identity, or the social psychology/sociology of work, so it leaves a lot of predictive/explanatory power on the table.
Many hypotheses to be tested!
New Substack post with @soumitrashukla9. Work has never been just about the income earned sans effort costs, as typical in economic models. Key aspects of people’s identity, their meaning, and sense of agency are all wrapped up with what they do for a living.
There has been a lot of conversation lately about how technological change, particularly AI, may disrupt this relationship, from Ezra Klein's column on his own mathematician father’s grapplings with losing meaning in mathematics, to Francis Fukuyama's post on AI, agency, and recognition. In this piece we take a historical perspective on how technological transformation has disrupted identity and meaning by affecting jobs, and how workers and society responded to these changes.
Much of the debate around AI and jobs has centered on wages and material consequences, but losing one's work is also intertwined with how people view their place in society. Most people know about the Luddites—skilled weavers and knitters—who protested the devaluation of their mastery. What’s less known is that the Luddites were actually experts in using the machines as well, and that their protests prompted prominent economists like David Ricardo to rethink the role of automation in society, and later inspired the Arts and Crafts movement. A century later, when recorded sound displaced 20,000 cinema musicians, their campaign against "canned music" and two-year recording ban pressured labels into funding live performance.
Soumitra and I have been working on versions of this essay for months. He led the piece, and we’re both really excited to get it out there.
Link to the post here: aleximas.substack.com/p/tech…
Also, there's wonderful work on the dynamics and consequences of such shifts (e.g., @mrfrank5790) for employment and wages. Part of the problem here is that it's hard to learn new skills for a new job, and the new jobs you *could* do are not geographically nearby.
Matt Beane retweeted
By every measure, Opus 4.8 beats human experts as peer reviewers in a study of 124 Harvard social science papers covering a range from statistical analyses to formal models to qualitative work.
As academics, we need to figure out how to adapt the practice of science to AI, fast.
The @lucasmuseum is wonderful. So are robots. And Lucas is a Rockwell fan. So, naturally:
Matt Beane retweeted
🧵1/10 Given the recent reactions of mathematicians over rapid advances in AI, I think it is important to remember that technological change has historically been a very personal event for people.
Over the history of technological progress, looms have fallen silent, switchboards have gone dark, and factory lines have shrunk bringing with them a human crisis of identity and what it means to be a master of one's craft.
Highest endorsement!
We are settling for an extractive playbook for industrializing knowledge work. The second-order consequences are vile: limits to progress and human potential.
I started to talk this way on the @TEDTalks stage in 2019 and started building a solution in 2022.
New essay in @ForeignAffairs with James Manyika and @elerianm on steps that can be taken to help AI deliver on its economic promise, drawing on lessons from cooperative game theory as a guide.
Most conversations treat AI as a general-purpose technology (GPT), and some compare it to the Industrial Revolution. But we think it is more than that: it is also a method for creating new inventions, like the microscope or the scientific method. Economist Zvi Griliches famously called such discoveries IMIs--inventions of a method of invention. Seen this way, AI carries the potential of the Industrial Revolution and the Enlightenment combined.
Yet realizing this potential is a coordination problem. Delivering on the promise of AI requires progress not just on furthering its capabilities, but on building out infrastructure, deep adoption across the economy, and buy-in from many different groups. This is unlikely to happen if people do not feel they have a stake in the upside of the technology. Promises of abstract scientific and medical breakthroughs in the future won't change this when anxieties about near-term disruption are running hot.
Rather than treating AI like a zero-sum game, we argue that it is useful to look to cooperative game theory as a guide. Because models, compute, organizational change, and human upskilling are strong complements, AI has the structure of a "convex game": Each group that joins makes everyone else's contributions more valuable. That makes it possible to form stable coalitions of workers, communities, firms, and developers where incentives are aligned and everyone has a stake in the gains.
The cooperative approach will of course not be easy, but we will make the case that coalitions can start small and grow as the mutual gains become tangible. We provide concrete examples of how this can be done in practice and at scale.
Link to essay here: foreignaffairs.com/guest-pas…
Claude now creates a unique kind of friction: paternalism.
Unilaterally redacts "sensitive" data without explanation. Every time, when queried, it offers no defense of such behaviors, says "that's not my call to make" and moves on. Harness config changes have no notable effect.
Congratulations, Ethan. We're very fortunate that you've ended up as the world's AI bard - warm, creative, wacky, eclectic, and practical.
Highly recommend!
My new book, Co-Existence, is out today. It is a guide to how to live & work in a world where AI is sometimes, but not always, better than you.
(And, for the release, I worked with Opus 5.5 to make a quite interesting experience for the book's website: co-existence.ai/)
Matt Beane retweeted
New in @PNASNews:
In two large experiments (~18k obs from ~14.8k people), we find that brief conversations with AI can get people to take real political action - incl. signing real petitions and giving money to charity.
Surprisingly, the persuasion strategy that worked best for changing people's attitudes - packing convos with info - was the least effective at changing behavior!
🧵
The fact that a transformer-based model trained on language and images can do next frame prediction AND estimate time to a predicted state utterly blows my mind.
This was viewed as impossible about six months ago (or perhaps less!).
ps: this is not the robotics bottleneck.
Two new egocentric benchmarks from @microagi evaluate VLM readiness for robotics:
MicroPlanBench and MicroAnnotationBench.
tl;dr: Semantic reasoning ≠ perceptual timing
A lot of work to do.
MicroPlanBench (consequential next-action planning)
When predicting the single valid next atomic action from 8s of egocentric video, SOTA models still fail ~1/3 of the time (GPT-6 Astra: 65.6%, GPT-6.1 Sol: 65.5% at 1/5th cost).
- GPT-6 Astra error breakdown: 60% wrong manipulation, 23% wrong hand assignment, 17% wrong object/destination.
- Across all flags, 42% of failures involve tracking the wrong hand.
microagi.ai/benchmarks/micro…
MicroAnnotationBench (continuous temporal grounding)
Knowing what happens ≠ knowing when
Continuous 60s+ dense manipulation video remains unsolved for generalist VLMs:
- GPT-6 Astra/Sol over-segment (~22 segments/clip vs. 13.8 human mean), collapsing precision (~39%) and strict Boundary F1 (<20%).
- Scaling Test-time compute fails to fix perceptual boundaries: scaling Gemini 4 Argon to high reasoning effort dropped its headline score by 4.9 points while costing 4.7x more
microagi.ai/benchmarks/micro…
Currently, Small specialists match/beat frontier APIs at scale.
@microagi's self-hosted nanoannot-1 outperforms the frontier models (Gemini 4 Argon) across the board:
- 54.8 vs 49.6 overall;
- 32.9 vs 27.3 on strict ± 0.667s Boundary F1
- ~1/10th the cost (~$0.30 vs $3.14/video hr & >300x cheaper than GPT 6 Astra)
Solid work by @ErguzelYag, Ömer Adanir & Team.
Matt Beane retweeted
On "where is the AI ROI", diffusion of every new technology is slow because of organizational redesign constraints, slow information spread, and the need to invent complements. To predict *future* economy wide ROI, you should look at *right tail* ROI among orgs today.
Matt Beane retweeted
Brandon and I pledged 50% of our equity. This was the principal reason that I left Google for Anthropic. The existence of this kind of program was a costly signal that Anthropic wasn’t just talk, at a time when Anthropic didn’t even have a released model. Fund a bright future!
We’re committing a portion of Anthropic’s shares to the Anthropic Public Benefit Fund, a new wholly owned entity dedicated entirely to the public benefit.
It will start off with 5% of our shares. For every 50% increase in the company’s value over a set baseline, we can contribute up to an extra 2%, subject to our board’s discretion.
In total, this means that roughly 20% of Anthropic’s shares are committed to the public good. Our co-founders have already pledged to donate 80% of their equity, and around half our employees are donating a portion of their equity as part of a donation matching program we’ve run since our founding.
In the post below, we share more details about our new fund.
anthropic.com/news/public-be…
Hot take: AI will help us solve quantum gravity sooner than scale multipurpose robotics.
And yes, that would mean getting humans to deploy new experiments with as-yet-uninvented technology.
Agree with Miles. Important viewing. I'm on the front lines, watching companies automate work with agents. The worker is getting left behind.
We can get results, enhance AI AND enhance human flourishing in one move.
It's time to do better. Right now.
Matt Beane retweeted
I've been seeing an increase in companies talk about how they have <coding agent> write all their evals automatically based on production traces
while i think this is a good start, it's very important to note that if you only ever create evals based on traces generated from your users, you'll only ever have "regression test evals".
sure, you should have some of these. but if you're doing eval dev, imo >50% of your time should be in identifying what your agent can in theory handle, and writing evals for that.
this will have overlap with what your users are asking of your agent in prod, but it'll also catch issues before your users can
you cant just turn prod traces into evals. you need a system which can also identify net-new or potentially lacking areas of your agent, and build evals there