Mathematician, trader, software engineer, cloud architect, ex-lawyer. Cloud Tech, Fintech, DevOps, AWS, Java & Spring, Go, Rust, Haskell, React, Scala.
Joined March 2012
- Tweets146
- Following1.5K
- Followers92
- Likes954
Been having an issue with Jazzer, FastJSON, and Netty (for a low-latency JVM WS Client) on JDK 27 (integrity changes). Sol and Astra tripped over the “security work” false positive check - third time’s the charm thought I, but 16h of Astra churning later, I was at square one again. It even suggested to fork Jazzer😅 Stoped looking plausible to me. So, I handed this task over to DeepSeek v4.1 Flash, and at turn-by-turn ~180-230 tps (with CoT/thinking prose) it fixed the issue for me systematically, though with a couple avoidable detours. At an average ~92tps for code generation and after 34 minutes unattended work, it actually managed to fix it. Saved me several hours of debugging but deprived me of a fun rabbit hole. Less fun but more time for what really matters. Now I only need to review it 🥲
GPT 6.1 Sol is such a weird model. Very opaque CoT updates, without a good harness no idea what it’s doing and why it’s taking so long. You really need tracing hooks to monitor it properly. There’s little native observability, outside of Codex app - you need a semantic/interpretation layer to keep track of it.
Did I say how slow it is? Very slow. Astra now at 26 tps, Sol has been at about 27 tps. It’s not so easy to harness Sol. Given how easy DeepSeek, GLM, and Claude are, not sure it’s worth the effort, really. Fast mode at x1.5 and ultrafast at x6 not really worth the extra token burn. If at least it were x3 at no extra cost…
I’ve just given it a simple ops task, something Claude Sonnet 5.5 does in three minutes, Sol has messed up my project data (wasn’t allowed to touch it), so I restored it and handed over to Astra - and it’s been churning for over half an hour now… still no result - at the price point of Claude Fable… erm…
The legacy x20 $200 ChatGPT Pro plan’s weekly allowance used up with Sol in merely 3 days - if it were three times as fast (as DeepSeek, Opus and Sonnet are), it would have been just one day - no Astra, just Sol. Now that it’s half that in 4 days, what’s the point of sticking to Codex?
Looking closer, beware of conflating the performance numbers. For example, my off-the-shelf harness of choice inflates the tps figure by including the generated prose which is meaningless, hence DeepSeek appears misleadingly so much faster in marginal throughput than Opus or GPT models. Opus generates terse prose by default and GPT-6 models are very sparing in prose, mostly tool calls - CoT encrypted - distillation protection. What counts is the total token count and total duration from task begin through task completion (acceptance gate passed), and then you take the uniform throughput over all turns, extracting the prose - it’s the target output that counts, not the intermediate chatter. Extra chatter only pollutes the context. DeepSeek is very chatty, like Opus 4.x. Harness/adapter + API/subscription endpoints + skill pack/LSP/MCP matter a lot. For realistic performance measurement and evals, you need a clean, controlled environment.
Nick retweeted
Anthropic Subscriptions Offer 5x+ More Value Than OpenAI
Limit testing every AI subscription plan from Anthropic, OpenAI, Meta, SpaceXAI, MiniMax, Moonshot, Zdotai, Cursor, and Cognition
newsletter.semianalysis.com/…
It’s really inexplicable how much my UX with Codex models has deteriorated over the past couple of months. And the GPT models increasingly misinterpret my prompts.
Most LLM benchmarks exclude the time a model/harness/adapter combo takes to complete the task - they ought to weight that highly - it’s real human time spent on waiting. On top of that, GPT models tend to have shorter memory and churn more on longer agentic tasks - there’s excessive churn burning tokens with shortest frontier context windows in the Codex GPT models (274k GPT tokens vs mostly (even if degrading) max 1m tokens in many other LLMs) - server-side compaction drops details that the model needs to rediscover, making it harder to converge and taking longer to stop at some result. Understood flipside of longer context windows is measurably degraded performance. In my experience still better with most of the popular models, including latest DeepSeek and Qwen.
Really, Codex models are no good for coding, even when it’s bare markdown docs about code. Sol 6.1 churning a single markdown doc for 3h now. Another session running for over 60h still not done. In both cases the output is garbage, each turn regresses on something. Fresh sessions aren’t helping. This is incredible. I’m keeping Codex models only for code review and for chat - they’re good at that. But alignment/SFT/RLHF for coding is really worse than ever. This is the same with the Codex vendor harness, pi, subscription and direct API calls. Exacerbated by short context windows, albeit no better when expanded per config.
Not only are they slow, they’re also not effective.
We often forget to talk about the speed the LLMs run, focusing on contrived task-oriented benchmarks.
Here are my observations over the past few days:
- GPT-6.1 Sol: 19.4 tok/s avg
- GPT-6 Astra: 33.5 tok/s avg
- Claude Opus 5.5: 102.3 tok/s avg
- DeepSeek 4.1 Flash: 179.1 t/s avg
- Grok 4.7: 74.6 t/s avg (Grok 4.6 reached ~180t/s in Cursor).
GPT still very slow. To come closer to Opus you need ”ultrafast” which is way more expensive.
On top of that GPT-6.1 Sol usually fails to converge in longer agentic sessions, aimlessly expanding the scope, churning tokens. Sol really wears you out until it completes the tasks c you need to cut them small for it. Also due to context window.
Nick retweeted
Can LMs discover & fix bugs if you don't tell them what went wrong? To proactively maintain a repo, agents need to find problems before users do.
SWE-sweep benchmarks this on 100 repos, 22 languages, 4k real bugs (numpy, php interpreter, lean kernel). Top models get <5%.
Nick retweeted
I’m very excited to share @nvidia Kumo Tabular, a new family of foundation models for tabular data.
Kumo Tabular establishes the new Pareto frontier across the entire accuracy–inference-time tradeoff.
Just as importantly, we are releasing it openly: open weights, open-source software, and a permissive license for commercial use.
HuggingFace: huggingface.co/nvidia/Kumo-T…
GitHub: github.com/NVIDIA/structured…
Nick retweeted
MiMo-V2.6-Pro debuts as the top open weights model on the Artificial Analysis Intelligence Index (46). At $0.13 per Intelligence Index task, it lands on the Intelligence vs. Cost per Task Pareto frontier
@Xiaomi has just released MiMo-V2.6-Pro, an open weights model with major advances in intelligence over its predecessor, MiMo-V2.5-Pro (Intelligence Index: 26). Despite the improvement, it retains the same attractive pricing at $0.435 per 1M input tokens (with a 99% cache-hit discount) and $0.87 per 1M output tokens. This makes MiMo-V2.6-Pro one of the most cost-efficient models to deploy.
MiMo-V2.6-Pro is an MoE model with 1.02T total parameters and 42B active parameters. Stay tuned for additional analysis of the model.
Check out MiMo-V2.6-Pro full benchmarking breakdown here: artificialanalysis.ai
A nice compilation of standard formulaic representations (and references) of many of the traditional instruments, statistics, and strategies (all referred to as “strategies” in that publication).
Mathematics for Trading — 151 Strategies (download open-access 361-page PDF): papers.ssrn.com/sol3/papers.…
Nick retweeted
🚀 Introducing DeepSeek-V4.1-Flash: smarter, faster, more efficient.
🔹 Introducing the smallest model in our new architecture family, with native visual understanding.
🔹 Designed for greater capability, faster inference, higher throughput, and scaling to larger models.
1/6
Nick retweeted
Hi Astra users. A reset and a quick update on quality issues that have been posted around.
Working with some of you, we have found and fixed the following issues:
- Some skills written for previous models were triggering too often or preventing the model from checking its work.
- An opt-in context management experiment that could cause early stops or replies to older messages. We've disabled it. Our rough estimate is that 4-5k users were affected by this experiment.
- We've also removed some badly configured engines that resulted in a measured quality degradation for a long tail of traffic flowing through them.
We’ve also made some more minor improvements and things should feel significantly better across the board. More consistent follow-through, better tracking of your latest message, and better checks on the work as it’s going through the motions.
The examples posted and all the users who worked directly with us were incredibly useful in helping fix things quickly. Always grateful for this incredible community.
And of course, a reset is also landing by midnight today.
More and more indications that existing skills and agents (and “harnesses”) need to be updated for the new models.
@OpenAIDevs have just shared this high-level overview: developers.openai.com/blog/r…
And @thsottiaux confirms there are some necessary updates in the skills/agents.
Existing SDD frameworks can become legacy overnight and only your token burn will tell. That’s why it’s important to read and review all your skills and vendor third-party skills/agents.
Hi Astra users. A reset and a quick update on quality issues that have been posted around.
Working with some of you, we have found and fixed the following issues:
- Some skills written for previous models were triggering too often or preventing the model from checking its work.
- An opt-in context management experiment that could cause early stops or replies to older messages. We've disabled it. Our rough estimate is that 4-5k users were affected by this experiment.
- We've also removed some badly configured engines that resulted in a measured quality degradation for a long tail of traffic flowing through them.
We’ve also made some more minor improvements and things should feel significantly better across the board. More consistent follow-through, better tracking of your latest message, and better checks on the work as it’s going through the motions.
The examples posted and all the users who worked directly with us were incredibly useful in helping fix things quickly. Always grateful for this incredible community.
And of course, a reset is also landing by midnight today.
Just experienced a weird @OpenAIDevs Codex usage glitch while running three GPT-6 Astra (medium) driven sessions. My 7d (94% and 5d9h remaining when started) got depleted somehow and half an hour later I’m at 12% and 1d14h remaining quota. Fable 5.1 manages to last way longer.
Looking at local optima churn with GPT-6 Astra and GPT-5.6 Sol as well as Fable 5.1, it’s time to revise our understanding of instructions such as “meticulous” and “rigorous” review, “impeccable” implementation, and similar. For the earlier models, it was often necessary to get them to an acceptable level, with the newer models it’s a hazard that forces them to loop around saddle points trying to squeeze the last bit of “quality”, forgetting about the overarching goals and structure of the workstream.