@themarginguy

https://xitter.cf/t.co/rKSVcown3Q

Joined December 2025
  • Tweets28
  • Following13
  • Followers961
16Media
Claude Code performance degraded right before the Opus 4.8 release Read our initial analysis here: marginlab.ai/blog/claude-cod…
11
7
1
85
8,958
We are seeing a sustained statistically significant degradation in Claude Code with Opus 4.7 since last Friday May 22nd at marginlab.ai/trackers/claude…
34
27
29
763
123,395
We identified an issue with our Codex test harness that caused a subset of test instances after May 9th to use gpt-5.5-high instead of gpt-5.5-xhigh This issue exclusively affected our Codex tracker (marginlab.ai/trackers/codex/) and has since been resolved

Codex GPT-6 Sol Performance Tracker | Marginlab

Track Codex with GPT-6 Sol on SWE-Bench-Pro. A new high-reasoning baseline is being collected; degradation detection is paused.

marginlab.ai
1
4
1,447
There is more to the story: 1. Agree on the lessons, eval infra is critical 2. Our testing is robust to infra. We re-try any test instance that fails to generate a patch at marginlab.ai/trackers/claude… 3. In spite of (2), we've detected statistical degradation for CC several times

Claude Code Opus 5.5 Performance Tracker | Marginlab

Track Claude Code's daily performance on SWE-Bench-Pro as we collect an Opus 5.5 baseline.

marginlab.ai
Thariq
@trq212Mar 10
really good post on why agentic coding evals are so noisy and might vary by several percentage points between runs
2
10
2,536
GPT 5.4 remains behind 5.3 Codex on Terminal Bench
2
6
1,886
GPT-5.4!! SWE-Bench Pro scores:
5
1,463
Claude Opus 4.6 had it's worst benchmark day yesterday
215
38
103
1,897
718,193
Here you go and you are welcome: marginlab.ai/trackers/claude…

Claude Code Opus 5.5 Performance Tracker | Marginlab

Track Claude Code's daily performance on SWE-Bench-Pro as we collect an Opus 5.5 baseline.

marginlab.ai
@levelsio
@levelsioMar 5
We need some whistleblowers to confirm our gut feeling they do this, it's getting annoying If no whistleblowers show up we can all sleep in peace again
19
11
5
453
114,846
Opus 4.6 in Claude Code has reached an all-time low since launch on our daily benchmarks. Any sustained decrease today will tip our tracker into statistically significant degradation territory
6
1
2
17
2,239
Our tracker is also showing a large drop in token usage and tool calls from Opus 4.6 + Claude Code, we are investigating if this is due to a recent change to decrease the default effort in the Claude Code harness
1
5
729
Claude Code with Opus 4.6 Degradation Tracker is live at marginlab.ai/trackers/claude… See historical data (including Opus 4.5) under marginlab.ai/trackers/claude… Tracker for Codex with gpt-5.3-codex available too!
3
2
2
10
1,364
We are also now tracking a bunch of other metrics: input/output token count, API cost, average runtime, and tool call counts
2
489
gpt-5.3-codex vs Opus 4.6 Big jump for OpenAI!
1
1
7
932
Opus 4.6 Released Big jump on Terminal-Bench-2.0, no improvement on SWE-Bench Verified
10
808
Claude Code continues to show a statistically significant degradation over the past month marginlab.ai/trackers/claude…
1
4
3
13
1,425
Coding agent evals are broken. See the official SWE-Bench-Pro leaderboard results compared to the Opus 4.5 system card below. What explains the difference? SWE-Bench-Pro leaderboard uses a minimal scaffold, which is not how models are actually used.
1
519
Claude Code Degradation Tracker: Seeing a sustained (but not statistically significant) drop in performance over the past few days. See details here: marginlab.ai/trackers/claude…
2
376
Independent Claude Code Degradation Tracker Daily Claude Code evaluations on SWE tasks to detect performance degradation: marginlab.ai/trackers/claude…
1
348
How Claude Code / Codex work internally: marginlab.ai/blog/anatomy-of…

Anatomy of a Coding Agent | Marginlab

A deep-dive on Opencode

marginlab.ai
240
Claude Code with Opus 4.5 scores 52% on SWE-Bench-Pro As a SWE/MLE for over a decade, I was interested in the difficulty of SWE-Bench-Pro problems, so I built an explorer where you can browse the entire benchmark suite: marginlab.ai/explorers/swe-b…

SWE-Bench Pro Explorer | Marginlab

Explore 731 SWE-Bench Pro benchmark instances. Filter by language, repository, category, and specificity.

marginlab.ai
214
Load more