OpsConfig retweeted
Narrator: he was not, in fact, allowed to say “I’m quite unhappy with much of what OpenAI does.”
I’m quite unhappy with much of what OpenAI does.
I am very happy that I’m allowed to say “I’m quite unhappy with much of what OpenAI does.”
Readers added context they thought people might want to know
Korbak was fired ~3 weeks later. He says for safety concerns/METR talks (his job); OpenAI says mishandling sensitive info, not for speaking out.
x.com/tomekkorbak/st…
x.com/OpenAINewsroom…
OpsConfig retweeted
Getting to watch these beagles from Marshall Bioresources run on the grass for the first time was pure joy!
#beagles #dogsofx #marshallfarms #labdogs
OpsConfig retweeted
New Paper: Turns out that LLM think traces don't necessarily have any end-user semantics even on benchmarks specifically designed to showcase such semantics.. 🤷♂️
As some of y'all know, we have been looking at the question of end-user interpretable semantics for the reasoning models for a while (c.f. arxiv.org/abs/2504.09762 (#ICML2026),
arxiv.org/abs/2505.13792 (#ACL2026) &
arxiv.org/abs/2505.13775 (#TMLR➡️#NeurIPS2026))
One question I get asked is whether that lack of semantics just an artifact of the benchmarks *we* made.
What if we look at benchmarks that were specifically proposed to show off the semantics of thinking traces?
As it happens, earlier work by AllenZhu @ZeyuanAllenZhu has indeed created such a benchmark, iGSM, and made claims about intermediate token semantics based on it.
Since AllenZhu's work on "Physics of LLMs"--that is partly based on their analysis on iGSM was influential, we wondered whether they found conditions under which traces do indeed have semantics.
Turns out, (perhaps unfortunately for the "please let thinking traces have semantics" contingent), that iGSM traces don't have semantics either, and models trained with swapped/corrupted traces can do well too. The results seem quite in line with--rather than being contrary to--the position we have taken in our earlier work.
Read the details of this very crisp work, lead by @ratishsp and Pranabendu, in the following thread.👇 – at Tempe, AZ
Happy to share our paper, "Correct Answers, Invalid Traces: What Verifiable Grade-School Math Reveals About Chain-of-Thought Traces".
When a model gets the right answer, is the reasoning it produces also right? We test this on synthetic grade-school math where every step of a trace can be checked by a program.
Thread below.
OpsConfig retweeted

The Yale Review@yalereviewSep 16
🇺🇸
United StatesConnected via United States App StoreAccount-level information, not a live location or per-post device.
Annie Dillard has died. In our Summer 1988 Issue, she wrote about the "surprising letters" and gifts that fans and readers sent her over the years—a fossil fish, a marriage proposal, a check for $2.50, and, again and again, questions about God. yalereview.org/article/annie…
OpsConfig retweeted
Breaking News: Annie Dillard, the Pulitzer Prize-winning author of the modern classic “Pilgrim at Tinker Creek,” has died at 81. nyti.ms/4hrp9BD
OpsConfig retweeted
A lovely @nytimes obit for my brother who died on Labor Day.
Jonathan Cooper, Who Made Violins the Old-Fashioned Way, Dies at 76 nytimes.com/2026/09/12/arts/… via @NYTimes
OpsConfig retweeted
Two heroes of today's Navier-Stokes story are not getting enough credit. Córdoba and Martínez-Zoroa came up with the crucial strategy. Others built on it and, with the help of AI, provided the computations that finished the problem. Thankfully @QuantaMagazine gets this right:
OpsConfig retweeted
What an incredible way to say "Your usage limits will decrease by ~18%"
Starting September 14, we're permanently raising standard weekly limits in Claude Code by 25% for Pro, Max, Team, and seat-based Enterprise plans. Until then, the current 50% increase will be in place.
Readers added context they thought people might want to know
While standard base limits are increasing by 25%, this change represents a net 17% reduction in weekly limits compared to the active 50% promotional limits users currently experience, as confirmed by the official Claude account: x.com/claudedevs/sta…
OpsConfig retweeted
Comments such as these are instructive. If you have some belief about frontier labs, but frontier lab employees act inconsistently with this belief, what is the correct Bayesian update?
Replying to @andrewho03
The fact that you chose to start a data company despite what you saw at OAI baffles me. You no longer have access to the best models to use to create the data. Is that not crippling?
OpsConfig retweeted
One of the Keen researchers asked for guidance on compute expenditures. I replied: “Visualize the stack of $100 bills being set on fire with each run and weigh it against the knowledge sought.” In many ways, research today is in a better position to turn money into insight than ever before, but it can still be squandered.
OpsConfig retweeted
I'm actually fairly bearish on frontier lab valuations. I've never seen the reasons articulated to my satisfaction, so before I go to sleep, I wanted to quickly jot down my thinking here.
The basic issue is that the labs are highly unprofitable. This may seem like a simple point, but private market valuations can be relatively irrational; however, like with $SPCX, post-IPO pricing will likely be much more punishing, especially as the standard 6-month lockup period expires and selling pressure intensifies.
Many people claim that the labs have high margins. Yet even with high margins, a valuation of $1T would be justified only if the labs were doing nothing aside from serving inference (thus reducing costs only to those relevant to inference) and posting annual revenue numbers in the $100-200 billion range assuming ~80% gross margin and a 20x earnings multiple.
This assumption is obviously not true, because the frontier labs have to continually spend money training the next generation of models. This is because of market competition from runner-up firms. For example, if OpenAI had paused model development last year, there would no longer be any point in paying GPT-5 API prices when you can just use Qwen or Kimi instead for much cheaper. Thus, the labs are forced to invest ever-increasing amounts of money in model training, in a way such that at any given point of time, the amount you're forced to invest in the next model is dramatically higher than the amount of money you're actually making, because even if your revenue goes up with higher model capabilities, so do your future training costs. This is a profoundly punishing dynamic which severely penalizes frontrunners.
(There is also a related subpoint where frontier labs claim they can distill their leading models to win out at lower intelligence levels as well. This makes no sense because the revenue numbers involved are far too low when taking into consideration the rather low margin of such inference.)
Frontier lab valuations appear largely to be based on the assumption that as you scale up, the capabilities which emerge will be sufficiently general and profound that we'll see explosive growth (epoch.ai/publications/explos…) from things akin to AI agents starting and autonomously managing entire companies of subagents. But it's not clear to me that this is the case; indeed, as I mentioned in my previous post (xitter.cf/andrewho03/status/2082…), I believe that capabilities growth will be slower, spikier, and more data-limited than people currently assume. It may be the case that eventually we will see explosive growth of this nature with full automation of the economy, but at the very least my viewpoint implies much longer (multi-decade) timelines until we reach this point. It is not clear to me that the frontier labs will be able to operate unprofitably for so long, although I suppose maybe this foreshadows some sort of inevitable nationalization.
I also want to make a broader point about technological diffusion. The reason why technological diffusion is slow isn't just because, e.g., old people take a long time to learn how to use technology (although this is of course a contributing factor to some degree). In my view, it's because when a new, revolutionary technology comes along, the ways to incorporate that technology into subsequent developments are not always obvious, and in fact they cannot necessarily be arrived at through the application of pure reason. If they could be, then perhaps frontier models, at a certain point, would have a perfect understanding of how the LLM application layer should be developed, and they would then autonomously code, deploy, and sell such a layer.
But it seems more plausible to me that this diffusion is limited moreso by the hard problem of economic calculation--that is to say, the Hayekian notion through which the price system gradually promotes efficient allocation of resources and which cannot be simulated through central planning--and that even if we froze current capability levels at today's levels, it would take well over two decades to fully integrate in LLMs into our lives. Such a view is consequently rather bearish for the continued profitability of labs as it reduces their prospects for finding, say, something else comparable in profitability to coding agents, which seems to have been a somewhat lucky discovery by Anthropic to begin with. That is to say, even if you spam FDEs you aren't necessarily going to be able to just figure out the "correct" product shapes fast enough.
Overall, I don't think that people have clearly reasoned through their mental models for why lab equity should be worth as much as it currently is, and that if you actually bother to write down such a model, you may not arrive at the conclusion that you want to arrive at. This isn't to say that I don't expect AI to experience a huge (industry-wide) boom in the coming decades, but just that I'm not entirely sure I would buy OpenAI or Anthropic stock at latest valuations if I were given the opportunity to do so.
Of course, as an ex-lab employee, arguably this is talking against my own book; I should really be giving people more reasons to be bullish. But in the end, my influence is so small that it doesn't make a difference, so why not have some fun?
New blog post on what would be true about the world if trendline continues and leading lab hits $1T in revenue by the end of next year.
In other words, why compute might get 10x+ more expensive in coming years
dwarkesh.com/p/why-compute-m…
OpsConfig retweeted
It’s very hard to distinguish AI security incidents from AI marketing, and that’s actually a big problem going forward.
OpsConfig retweeted
Muse Spark 1.1 is trained on the screen recordings of Meta employees, which makes it best-in-class at applying for jobs at Anthropic
OpsConfig retweeted
This is cool
Replying to @TheStalwart
I was curious so I onboarded GLM-5.2 and Opus 4.8 to a chess eval app I have been working on:
youtu.be/z9DytGyAr-0?si=x_Sf…
I am passing board position and legal moves at each turn, but you still get a sense of the model's relative capabilities.
OpsConfig retweeted
Replying to @NielsRogge @OfirPress
Contamination, irrelevant numbers.
xitter.cf/JFPuget/status/2067328…
xitter.cf/JFPuget/status/2067509…
Replying to @Math_MntnrHZ
Exactly. After our NVIDIA team won swe-bench verified we stopped when o3 came out. It was identifying the right files to edit without opening them...
This benchmark is a joke.
Not the idea of it, it is a great idea. But it is contaminated.
Better look at variants where the test set is made of recent commits like swe-bench pro or swe-bench live. Curiously, frontier labs don't publish results on these unless mistaken.
OpsConfig retweeted
BTW, this is what happens in swe-bench verified.
Indeed, swe-bench test data is available as commits in github repos. Any frontier llm has seen these when training.
Numbers reported on swe-bench are not reliable.
OpsConfig retweeted
While working on the CRC of our #ICML2026 position against anthropomorphization of LLM intermediate tokens, I can't stop chuckling re-reading my rebuttal to the reviewer who said that my sentences in the submitted version appear "overly AI-generated".. 😄 – at Tempe, AZ
OpsConfig retweeted
Nobody Cracks Open a Programming Book Anymore:
🔗unix.foo/posts/nobody-cracks…





























