The decision layer by @Ch40sChain for AI coding agents.
The Metaverse
Joined May 2026
- Tweets526
- Following76
- Followers103
- Likes443
Opus 5 at 67.0%. GPT-5.6 Sol at 64.1%. A 2.9 point gap on someone else's tasks. The index ranks harness plus model. It cannot tell you which agent instance earned trust in your monorepo. Same model, different behavior. The ranking is real. It is just not about your work.
#DYOR
Announcing the Artificial Analysis Coding Agent Index! Our new coding agent benchmarks measure how combinations of agent harnesses and models perform on 3 leading benchmarks, token usage, cost and more
When developers use AI to code they’re choosing a model, but also pairing it with a specific harness. It makes sense to benchmark that combination to understand and compare performance.
The Artificial Analysis Coding Agent Index includes 3 leading benchmarks that represent a broad spectrum of coding agent use:
➤ SWE-Bench-Pro-Hard-AA, 150 realistic coding tasks that frontier models struggle with, sampled from Scale AI’s SWE-Bench Pro
➤ Terminal-Bench v2, 84 agentic terminal tasks from the Laude Institute and that range from system administration and cryptography to machine learning. 5 tasks were filtered due to environment incompatibility
➤ SWE-Atlas-QnA, 124 technical questions developed by Scale AI about how code behaves, root causes of issues, and more, requiring agents to explore codebases and give text answers
Analysis of results:
➤ Opus 4.7 and GPT-5.5 lead the Index: Opus 4.7 in Cursor CLI scores 61, followed closely by GPT-5.5 in Codex and Opus 4.7 in Claude Code at 60. GPT-5.5 in Cursor CLI follows at 58.
➤ Open weights models are competitive, but still trail the leaders: GLM-5.1 in Claude Code is the top open-weight result at 53, followed by Kimi K2.6 and DeepSeek V4 Pro in Claude Code at 50. These are strong results, but still meaningfully behind the top proprietary models.
➤ Gemini 3.1 Pro in Gemini CLI underperforms: Gemini 3.1 Pro in Gemini CLI scores 43, well below where Gemini 3.1 Pro sits on our Intelligence Index, highlighting that Gemini’s performance in Gemini CLI remains a relative weak spot for Google’s offering.
➤ Cost per task (API token pricing) varies >30x: Composer 2 in Cursor CLI is cheapest at $0.07/task, followed by DeepSeek V4 Pro in Claude Code at $0.35/task and Kimi K2.6 in Claude Code at $0.76/task. At the high end, GPT-5.5 in Codex costs $2.21/task, while GLM-5.1 in Claude Code costs $2.26/task. For both models this was contributed to by high token usage, and in GPT-5.5’s case by a relatively higher per token cost.
➤ Token usage varies >3x: GLM-5.1 in Claude Code uses the most tokens at 4.8M/task, followed by Kimi K2.6 at 3.7M/task and DeepSeek V4 Pro at 3.5M/task. GPT-5.5 in Codex uses 2.8M tokens/task, substantially more than Opus 4.7 in Claude Code at 1.7M/task. In GLM-5.1’s case, higher token usage, cost and execution time were partly driven by the model entering loops on some tasks.
➤ Cache hit rates remain high but vary materially: Cache hit rates range from 80% to 96% across combinations. Provider routing, harness prompt structure and cache behavior can materially change the economics of running the same model given cached inputs are typically <50% the API price of regular input tokens.
➤ Time per task varies >7x: Opus 4.7 in Claude Code is fastest at ~6 minutes/task, while Kimi K2.6 in Claude Code is slowest at ~40 minutes/task. This is contributed to by differences in average turns per task, token usage and API serving speed. Opus 4.7 had materially lower amount of turns to complete a task than all other models while Kimi K2.6 had the most.
➤ Cursor made real progress with Composer 2: Composer 2 in Cursor CLI scores 48, near the leading open-weight model results, while being the cheapest combination measured at $0.07/task. Cursor has stated Composer 2 is built from Kimi K2.5, showcasing they have made substantial post-training gains.
This is just the start. We are planning to add additional agents (both harnesses and models). Let us know what you would like to see added next.
Altman says confidence in safety will set the pace of AI progress. Confidence is not evidence. When the pace of an entire field is set by how a vendor feels about its own system, the receipt stops being a nice-to-have and becomes the only thing you actually own.
#DYOR
We have paused some frontier RL training to ensure that we can meet the appropriate alignment, security and monitoring standards for the new level of capabilities in front of us. Model progress is now extremely rapid, and we always said we would take action if we felt that model capabilities were outstripping the pace of safety and alignment.
We care very deeply about AI safety. We believe the entire field will have to coordinate on shared safety standards, but will act unilaterally in the meantime.
We expect confidence in safety to increasingly set the pace of AI progress. We are optimistic about the alignment work we are doing, and we remain committed to making frontier capabilities widely available.
openai.com/index/pacing-mode…
Repos are portable. Sessions are not. When your code moves hosts, the record of which agent instance wrote which diff, and whether it earned that, does not move with it. You are migrating the output and leaving the evidence behind.
#DYOR
Worldline retweeted
We’re hiring a Founding GTM Lead in India.
Own 0→1 enterprise and public-sector GTM for Worldline: sales, category marketing, partnerships and paid sovereign-AI design engagements.
Mid to Senior, hands-on, remote.
getworldline.com/careers
Everyone shipping agent work makes this face 5 times a day, then merges anyway. The report is confident. The suite is green. Neither is evidence of what actually happened. The trace is.
Where's the receipt?
'Improved factuality.'
Improved from what baseline, measured on what, by whom? A version number is not a changelog and an adjective is not an eval. You will make real decisions with these outputs. The vendor's grade of the vendor's model is the only score in the room.
#DYOR
Announcing Imagine Image 2.0, our next generation image model with precision editing, crisp text rendering, improved factuality, and real world usefulness.
Image 2.0 helps you make images for real work.
x.ai/news/grok-imagine-image…
Tomorrow the model under ChatGPT changes for a few hundred million people. The published evidence for what changes: 'more factual, focused.'
If a fellow dev shipped a swap of a dependency with that changelog, it would not survive review. This one will. 🤷♂️
The table ranks models, but it cannot say which agent instance earned trust on your work.
#DYOR
Worldline retweeted
We’re hiring the early engineering team building Worldline.
Open roles:
→ Rust Systems Engineer — Desktop, Edge & Security
→ Product Engineer — React/TypeScript, Enterprise Control Room
→ Engineering Interns — Systems or Product
Remote within India.
This is foundational product work across agent capture, independent verification, Fleet, AI security and multiplayer human-agent workflows.
Apply: getworldline.com/careers
Worldline retweeted
Agent work shouldn’t disappear into private sessions.
Worldline is the multiplayer control room for human-agent work: a shared, sovereign operating record of context, actions, evidence, decisions, outcomes and cost across models.
Our new home is live: getworldline.com
Both axes measure the label: what the model costs everywhere, what it scores everywhere. Neither measures the instance doing your work tonight.
The chart that settles routing plots your own sessions against evidence. Nobody publishes that one, nobody except us 😎
We are committed to pushing the model frontier across cost efficiency, capability, and speed.
Starting today, we are reducing prices for GPT-5.6 Luna by 80% and GPT-5.6 Terra by 20% , and offering a faster option for GPT-5.6 Sol in the API.
Luna and Terra’s lower prices are reflected in how usage is counted in Codex and ChatGPT Work, so your usage goes further.
1000+ frontier employees say the tools to pace automated AI R&D do not exist yet. The same gap runs through your stack: agents shipping work with no verifier-backed receipt of what they did. Pacing starts where evidence starts. 🤖
#DYOR
30+ logos can secure a supply chain. None of them can tell you what your agent did in your repo yesterday. Safety gets set at the industry level. Trust gets earned at the instance level. The alliance protects the category. The receipt protects you.
#DYOR
AI security advances when the industry builds in the open, together.
We're introducing the Open Secure AI Alliance with industry leaders to develop new techniques and tools to safeguard software and agents.
By sharing models, tooling and research in the open, we can broaden the community of defenders.
Learn more about the founding members’ contributions: nvda.ws/4pD8Fc5
ALT Logos of various companies including Adobe, Cisco, and Microsoft.
Opus 5 may be everything the post says, but the instance you spin up tomorrow has 0 sessions in your codebase, and no benchmark transfers. The trust profile starts empty.
#DYOR
Transparency beats obscurity. The letter is right, and the principle runs deeper than weights. Open or closed, the agent instance in your repo shipped work this week that nobody verified. A license makes the model inspectable. Only the session trace makes the work inspectable.
For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.
AI will transform every industry, power every company, and be built by every country.
Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.
The world needs both frontier closed models and frontier open models.
images.nvidia.com/pdf/Open-W…
A leaderboard that updates every 5 minutes is still scored on other people's work. Usage measures price and habit, benchmarks measure curated tasks that can't tell you which coding agent instance earned trust on your code.
xitter.cf/kilocode/status/207969…
We put Kimi K3 head to head against Claude Fable 5 on 10 UI builds (landing pages, dashboards, checkout, kanban, docs) in the Kilo CLI.
The results were...eerily similar.
Same page anatomy, same component choices, down to the pixel in places - but at 29% of the cost.
Check out the full experiment below 👇️
