This is my own held-out test set, entirely distinct from anything you'll find in online evals. It's a small set of tasks, so it's useful only as a sanity/smoke-test. For anything that's close, there's a good deal of noise. For example, in my tests outside of this, DS4.1-1017 seems to edge out GLM 5.3 Flash on the hardest tasks at the higest effort levels. But GLM 5.3 Flash definitely does have a consistent place in the middle of the Pareto frontier for getting "fairly easy but not quite trivial" tasks done relatively quickly.

Seriously, stop wasting your time on social media and go do something out in the real world that doesn't require literacy and thinking. What you're doing here just ain't working at all.

I think this represents the Chinese labs taking different approaches to the pressures of distillation. Anthropic and OpenAI made it challenging to distill reasoning in a number of ways, including e.g. summarizing reasoning rather than displaying verbatim reasoning, and so on. I suspect that Zai responded to this in GLM 5.3 primarily by leveraging exploits that work on models ~1-2 generations behind, but have been patched on more recent models (e.g., the dumb trick of "just giving it an external reasoning tool" that it ends up plugging its "actual unsummarized reasoning into"). The "native unsummarized reasoning tokens" in SOTA models are truly "caveman-like", because the leading labs have spent a LOT of recent effort on "token efficiency". OTOH, I suspect that DeepSeek responded to this instead with a different strategy: They probably distilled more requests to the model to reason in prose with thinking off or low (and possibly settled for mixing in more thinking summaries from newer models, rather than settling for native reasoning from slightly older ones). This kind of "reasoning" is more verbose because it's an output trained for human eyes rather than being meant directly for model processing. The outcome of this could plausibly be a smarter model (since it's distilled from smarter models) that's more verbose because that's a tradeoff they decided when gathering the training data.

Wonder whether @helicone_ai / @mintlify is even searching for a breach.

Quite a regression compared to this. youtu.be/m0b_D2JgZgY?si=Z762…