Building https://xitter.cf/t.co/6ZEBsohKP9 | Compare LLMs across long context tests. Managing Member @ DeX Group LLC President @ Alkimi AI Corp Plus others.
New York
Joined August 2011
- Tweets506
- Following116
- Followers889
- Likes289
Context Arena: Added stealth model "Ox Alpha" (run via @OpenRouter) on 8-needle GDM-MRCRv2! Name will change on reveal.
The new stealth model hits 75.3% AUC@128k on high thinking, matching Gemini 3.1 Pro, GPT-5.6 Luna, Muse Spark 1.2, Grok 4.6, and GLM-5.3 (low) while beating Claude Sonnet 5. But max thinking actually hurts accuracy past 32k.
All results at: contextarena.ai
Highlights:
- Strong score: 75.3% AUC@128k on high thinking is close to Gemini 3.1 Pro (77.4% high) and ahead of GLM-5.3 (75.0% low), GPT-5.6 Luna (74.8% max), Muse Spark 1.2 (74.6% high), Qwen 3.7 Flash (74.2%), and Claude Sonnet 5 (71.6% max).
- More thinking hurts: max thinking scores lower than high at every length from 32k to 512k (like 70.1% vs 75.8% at 64k) while using ~34% more tokens.
- Holds up at 512k: scores 39.0% at 512k, staying ahead of GPT-5.6 Luna (37.6%), GLM-5.3 (36.2% low), Claude Sonnet 5 (32.0%), Gemini 3.1 Pro (31.1%), and Qwen 3.7 Flash (21.3%).
- First needle is easiest: needle #1 hits 87.2% accuracy while needle #6 drops to 42.7%. When it misses, 91.3% of mistakes pick another valid needle (usually the next one in line, seems to be a common bias for this model).
@ 128k (AUC / cumulative average):
- GLM-5.3 (high): 85.9% / 88.8%
- GLM-5.3 (max): 80.8% / 86.0%
- Gemini 3.1 Pro Preview (high): 77.4% / 85.1%
- Ox Alpha (high): 75.3% / 79.5%
- Pokee-Isaac (enabled): 75.1% / 82.5%
- GLM-5.3 (low): 75.0% / 79.6%
- GPT-5.6 Luna (max): 74.8% / 82.1%
- Muse Spark 1.2 (high): 74.6% / 81.8%
- Grok 4.6 (medium): 74.4% / 82.0%
- Qwen 3.7 Flash (enabled): 74.2% / 81.2%
- Ox Alpha (max): 71.6% / 77.9%
- Claude Sonnet 5 (max): 71.6% / 80.0%
- Ox Alpha (low): 69.1% / 71.1%
@ 512k:
- Muse Spark 1.2 (high): 44.5%
- GLM-5.3 (high): 41.9%
- GLM-5.3 (max): 39.3%
- Ox Alpha (high): 39.0%
- GPT-5.6 Luna (max): 37.6%
- GLM-5.3 (low): 36.2%
- Ox Alpha (max): 35.4%
- Ox Alpha (low): 34.8%
- Claude Sonnet 5 (max): 32.0%
- Gemini 3.1 Pro Preview (high): 31.1%
- Qwen 3.7 Flash (enabled): 21.3%
- Pokee-Isaac (enabled): 10.9%
These models, side by side: contextarena.ai/?needles=8&s…
Enjoy!
@OpenRouter @GoogleDeepMind @OpenAI @AnthropicAI @xai @Zai_org @Alibaba_Qwen @Pokee_AI
1/3
For those curious, here is the model release:
z.ai/blog/glm-5.3-flash
Name has been updated on ContextArena!
Accuracy at each length (low / high / max):
- 8k: 81.7% / 93.9% / 96.0%
- 16k: 70.8% / 79.3% / 82.6%
- 32k: 72.7% / 85.9% / 82.1%
- 64k: 74.9% / 75.8% / 70.1%
- 128k: 55.4% / 62.9% / 58.9%
- 256k: 60.1% / 48.7% / 55.6%
- 512k: 34.8% / 39.0% / 35.4%
Average output tokens through 128k:
- low: 69.1% AUC@128k, ~538 output tokens/question (~1,286 across full run)
- high: 75.3% AUC@128k, ~1,279 output tokens/question (~2,046 across full run)
- max: 71.6% AUC@128k, ~1,715 output tokens/question (~2,277 across full run)
Takeaways:
- Best setting: high. It gets the highest score (75.3% AUC@128k and 39.0% at 512k) while being faster and using fewer tokens than max.
- Max thinking hurts: max uses ~34% more tokens than high, but scores worse at 32k, 64k, 128k, and 512k.
- Low thinking wins at 256k: low starts slower at short lengths (81.7% vs 93.9% at 8k), but reaches 60.1% at 256k, beating both high (48.7%) and max (55.6%) with way fewer tokens.
Note: Thinking tokens are included in the output token count, so the actual answers are much shorter than the token counts show.
2/3
How it makes mistakes (on high thinking):
- 91.3% of mistakes (295/323) picked another valid needle in the text instead of making something up.
- Almost never broke format (99.8% followed the hash answer format).
Position takeaways:
- Needle #1 is easiest: finding the 1st needle succeeded 87.2% of the time, compared to ~59% for needles #2 through #8.
- Front of the text is easier: needles placed in the first 20% of the text scored 78.9% vs 53.1% in the last 20%.
- Mistakes overshoot: when missing, the model was nearly 3x more likely to pick a needle after the target (+1 needle: 16.9%) than before it (-1 needle: 6.1%).
- Middle needles are hardest: needle #6 had the lowest accuracy at 42.7%.
- No boost at the end: needle #8 was no easier to find than the middle needles (+1.6 points).
Full analysis: contextarena.ai/variant-bias…
3/3
Context Arena: Added @Zai_org's GLM 5.3 on 8-needle GDM-MRCRv2!
With high thinking on, GLM 5.3 jumps more than 20 points over GLM 5.2, scoring 85.9% AUC@128k (88.8% cumulative average). It holds above 89% accuracy up to 64k and lands right near Claude Opus 4.8 (87.8%) and Gemini 3.6 Flash (84.2%), but drops off past 64k.
All results at: contextarena.ai
Highlights:
- High thinking is the sweet spot: 85.9% AUC@128k vs 80.8% on max and 75.0% on low, while using less than half the thinking tokens of max (730 vs 1,668 per test).
- Big jump over GLM 5.2: beats GLM 5.2 max at every single length, gaining +20.5 points on AUC@128k (85.9% vs 65.4%).
- Strong up to 64k: 96.0% at 8k, 94.9% at 16k, 94.3% at 32k, and 89.6% at 64k, before dropping to 69.3% at 128k and 41.9% at 512k.
- Solid peer standing: right alongside Claude Opus 4.8 (87.8%), Gemini 3.6 Flash (84.2%), and DeepSeek V4 Pro 0813 (81.1%), and edges Claude Opus 4.8 at 512k (41.9% vs 39.8%).
@ 128k (AUC / cumulative average):
- Claude Opus 5 (max): 97.5% / 97.8%
- GPT-5.6 Sol (max): 97.1% / 97.9%
- Qwen 3.8 Max (xhigh): 96.1% / 96.9%
- Gemini 3.7 Flash (high): 93.9% / 96.0%
- GPT-5.6 Terra (max): 89.4% / 92.6%
- Claude Opus 4.8 (max): 87.8% / 90.5%
- GLM 5.3 (high): 85.9% / 88.8%
- Gemini 3.6 Flash (high): 84.2% / 89.3%
- DeepSeek V4 Pro 0813 (high): 81.1% / 86.9%
- GLM 5.3 (max): 80.8% / 86.0%
- Claude Sonnet 4.6 (medium): 80.0% / 83.0%
- GLM 5.3 (low): 75.0% / 79.6%
- Grok 4.6 (medium): 74.4% / 82.0%
- GLM 5.2 (max): 65.4% / 73.3%
@ 512k (at each length):
- Gemini 3.7 Flash (high): 77.7%
- Gemini 3.6 Flash (high): 61.1%
- GPT-5.6 Terra (max): 49.8%
- Claude Sonnet 4.6 (medium): 43.7%
- Claude Opus 5 (max): 42.5%
- GLM 5.3 (high): 41.9%
- Claude Opus 4.8 (max): 39.8%
- GLM 5.3 (max): 39.3%
- GLM 5.3 (low): 36.2%
- GLM 5.2 (max): 34.9%
- DeepSeek V4 Pro 0813 (high): 33.4%
- Qwen 3.8 Max (xhigh): 29.7%
These models, side by side: contextarena.ai/?needles=8&s…
Enjoy!
@Zai_org @OpenAI @AnthropicAI @GoogleDeepMind @deepseek_ai @Alibaba_Qwen @xai
1/3
GLM 5.3 high vs GLM 5.2 max, at each length:
- 8k: 96.0% / 95.1%
- 16k: 94.9% / 81.8%
- 32k: 94.3% / 74.7%
- 64k: 89.6% / 63.4%
- 128k: 69.3% / 51.6%
- 256k: 55.8% / 46.5%
- 512k: 41.9% / 34.9%
GLM 5.3 beats GLM 5.2 across every single context length, with the biggest jumps in the middle: +26.2 points at 64k, +19.7 points at 32k, and +17.7 points at 128k.
Reasoning tiers:
- max: 80.8% AUC@128k, 1,668 thinking tokens per test through 128k
- high: 85.9% AUC@128k, 730 thinking tokens per test through 128k
- low: 75.0% AUC@128k, 171 thinking tokens per test through 128k
Recommendation: use high. It beats max by 5.1 points on AUC@128k while using less than half the thinking tokens. Low thinking is fast and uses very few tokens, but drops 10.9 points behind high.
Main caveat: accuracy falls off past 64k. It stays above 89% up to 64k, but drops to 69.3% at 128k, 55.8% at 256k, and 41.9% at 512k.
@Zai_org
2/3
Looking at mistakes across all 861 tests on high thinking: when the model missed, 92.4% of the time (232/251) it picked another valid version of the requested needle rather than an unrelated one. Output formatting was followed 98.6% of the time (849/861).
Position takeaways:
- Asking for the 1st needle is much easier: needle #1 was found 92.3% of the time, about 24 points higher than needles #2 through #8 (56.4% to 81.5%).
- Earlier in the text is easier: needles in the first 20% of the text scored 87.1%, about 21 points higher than needles in the last 20% (65.6%).
- Slight bump for the last needle: needle #8 bounced back to 74.5% (vs 56.4% for needle #6).
Full analysis: contextarena.ai/variant-bias…
@Zai_org
3/3
Context Arena: Added @Alibaba_Qwen's Qwen3.8 27B on 8-needle GDM-MRCRv2!
Medium thinking scored best at 92.8% AUC@128k, finishing just 0.5 points behind Qwen 3.8 Max (medium). However, xhigh thinking performed worse: it used over 12x more thinking tokens per sample than medium, but scored 6.2 AUC points less than medium.
All results at: contextarena.ai
Highlights:
- Medium scores 92.8% AUC@128k (94.4% cumulative average). Low is right behind at 91.1% / 93.1%, while xhigh trails at 86.6% / 90.7%.
- At the full 128k length, low holds up best at 81.8%, followed by medium at 78.9% and xhigh at 74.3%.
- Medium closely matches Qwen 3.8 Max (medium) at every context length, staying within 1 to 2 points across all bins.
- Huge leap over the previous generation: Qwen 3.6 27B (thinking enabled) scored 74.5% AUC@128k, so Qwen3.8 27B gained +18.3 points.
- Turning thinking off drops AUC@128k to 43.1%, so keep thinking enabled.
- Recommendation: stick with medium for the best overall score. Avoid xhigh for similar long-context, which costs significantly more while scoring worse.
@ 128k (AUC / cumulative average):
- Gemini 3.7 Flash (high): 93.9% / 96.0%
- Qwen 3.8 Max (medium): 93.3% / 95.2%
- Qwen3.8 27B (medium): 92.8% / 94.4%
- GPT-5.5 (xhigh): 91.7% / 94.5%
- Qwen3.8 27B (low): 91.1% / 93.1%
- Qwen 3.8 Max (low): 91.0% / 93.1%
- GPT-5.6 Terra (max): 89.4% / 92.6%
- Qwen3.8 27B (xhigh): 86.6% / 90.7%
- Qwen 3.6 27B (enabled): 74.5% / 82.3%
- Qwen3.8 27B (none): 43.1% / 46.7%
@ 128k (pointwise):
- Gemini 3.7 Flash (high): 88.6%
- Qwen 3.8 Max (medium): 87.3%
- GPT-5.5 (xhigh): 85.8%
- Qwen3.8 27B (low): 81.8%
- Qwen3.8 27B (medium): 78.9%
- Qwen3.8 27B (xhigh): 74.3%
- Qwen 3.6 27B (enabled): 56.6%
- Qwen3.8 27B (none): 34.7%
These models, side by side: contextarena.ai/?needles=8&s…
Enjoy!
@Alibaba_Qwen @GoogleDeepMind @OpenAI
1/2
Qwen3.8 27B, bin by bin (medium / low / xhigh / thinking off):
- 8k: 99.3% / 99.3% / 100.0% / 60.3%
- 16k: 99.2% / 95.6% / 99.0% / 44.1%
- 32k: 96.7% / 95.3% / 92.9% / 51.5%
- 64k: 97.9% / 93.3% / 87.4% / 42.9%
- 128k: 78.9% / 81.8% / 74.3% / 34.7%
Medium and low perform similarly across all context sizes. Medium has a slight edge on overall AUC (92.8% vs 91.1%), but low holds slightly higher accuracy at 128k (81.8% vs 78.9%).
Thinking tokens and estimated standard cost through 128k:
- medium: 92.8% AUC@128k, 568 thinking tokens/sample, $12.75
- low: 91.1% AUC@128k, 423 thinking tokens/sample, $12.53
- xhigh: 86.6% AUC@128k, 7,019 thinking tokens/sample, $22.15
- thinking off: 43.1% AUC@128k, 0 thinking tokens/sample, $11.86
All 484 samples completed with 0 sample errors across all four modes.
@Alibaba_Qwen
2/2
We only tested this model up to 128k due to available providers having ~256k context length (no space for outputs at that bin), and as a result nothing interesting of note on the "bias" analysis.
But if you are interested: contextarena.ai/variant-bias…
Context Arena: Added @Pokee_AI's Pokee-Isaac 28B v0 on 8-needle GDM-MRCRv2! I was very eager to test this model's long-context claims!
Pokee's launch report highlighted strong long-context performance over models like GPT-5.6 Luna, Gemini 3.5 Flash Lite, Nemotron 3 Super, and Claude Haiku 4.5. On their reported RULER and MRCRv2 scores, it looked like long-context remained almost flat.
On independent MRCR testing, Isaac delivers on that through 128k, scoring 75.1% AUC@128k and 60.2% at 128k. But past 128k, it flips: accuracy falls to 18.7% at 256k and 7.4% at 1M.
Not sure why. May be due to different test sets (Note: I'm assuming the discrepancy is Pokee is using OAI-MRCRv2, while I used the harder GDM-MRCRv2). As a result, I held off testing 2M, 4M, and 8M datasets.
All results at: contextarena.ai
Highlights:
- 75.1% AUC@128k (82.5% cumulative average), placing #30 on the 128k leaderboard.
- Strong short and medium context: holds above 89% accuracy up to 32k (98.8% at 8k, 92.7% at 16k, 89.3% at 32k) and reaches 60.2% at 128k.
- Outpaces several small models on GDM-MRCRv2 through 128k: beats Claude Haiku 4.5 (13.8% at 128k), Qwen 3.5 122B (34.7%), and Nemotron 3 Super (38.1%), while edging GPT-5.6 Luna (max, 58.4%) and Gemini 3.5 Flash Lite (high, 55.5%).
- Steep drop past 128k: accuracy drops to 18.7% at 256k, 10.9% at 512k, and 7.4% at 1M (22.2% AUC@1M).
@ 128k (AUC / cumulative average):
- Pokee-Isaac 28B v0 (enabled): 75.1% / 82.5%
- GPT-5.6 Luna (max): 74.8% / 82.1%
- Grok 4.6 (medium): 74.4% / 82.0%
- Qwen 3.7 Flash (enabled): 74.2% / 81.2%
- Claude Sonnet 5 (max): 71.6% / 80.0%
- Gemini 3.5 Flash Lite (high): 64.4% / 73.2%
- Qwen 3.5 122B (enabled): 57.4% / 69.9%
- Nemotron 3 Super (enabled): 48.8% / 49.6%
- Claude Haiku 4.5 (enabled): 24.3% / 37.8%
@ 256k (pointwise):
- Claude Sonnet 5 (max): 52.2%
- Grok 4.6 (medium): 46.8%
- GPT-5.6 Luna (max): 45.1%
- Qwen 3.7 Flash (enabled): 43.1%
- Gemini 3.5 Flash Lite (medium): 40.3%
- Gemini 3.5 Flash Lite (high): 37.6%
- Pokee-Isaac 28B v0 (enabled): 18.7%
NOTE: Claude Haiku 4.5, Nemotron 3 Super, and Qwen 3.5 122B runs were cap to 128k
These models, side by side: contextarena.ai/?needles=8&s…
Enjoy!
@Pokee_AI @OpenAI @GoogleDeepMind @AnthropicAI @Alibaba_Qwen @xai
1/3
Pokee-Isaac 28B v0 (enabled), at each length:
- 8k: 98.8%
- 16k: 92.7%
- 32k: 89.3%
- 64k: 71.5%
- 128k: 60.2%
- 256k: 18.7%
- 512k: 10.9%
- 1M: 7.4%
The self-reported vs measured contrast:
Pokee's technical report shows MRCR scores climbing to 60.7% at 256k and 74.3% at 512k, ahead of Luna and Gemini Flash Lite. On GDM-MRCRv2, Isaac matches Luna through 128k (60.2% vs 58.4%), but drops by 41.5 points at 256k (18.7% vs 45.1% for Luna and 40.3% for Gemini 3.5 Flash Lite medium), reaching 7.4% at 1M.
Notes:
- All 1,171 samples across the 8k to 1M bins completed with 0 sample errors.
- Estimated standard on-demand cost is $5.74 through 128k ($85.59 for the full run to 1M).
- Context window note: While Pokee advertises up to a 10M context window, testing was capped at 1M because accuracy had already dropped near 0%.
- Reasoning token metadata is not returned by the API, so thinking token breakdowns are unavailable.
@Pokee_AI
2/3
Analyzing mistake patterns across all 1,171 outputs from Pokee-Isaac 28B v0:
Mistake breakdown:
- Through 128k: 95.5% of errors (85/89) picked another valid variant of the requested needle rather than an unrelated one, and wrong-needle errors were rare (4/484).
- Past 128k: wrong needles (313) and unmatched outputs (141) surged as needle interference overwhelmed the model.
- Hash key: the target random hash was present in 99.9% of responses (1170/1171).
Position findings:
- Asking for the 1st needle is much easier: finding needle #1 was 98.4% accurate through 128k (53.2% overall), compared to ~79% for needles #2 through #8 through 128k (36.3% overall).
- Front of text advantage: needles placed in the first 20% of text scored ~22 points higher through 128k than needles placed in the last 20% (97.6% vs 75.5%).
- Overall position slope: accuracy steadily declines as the target needle appears deeper in the context.
Full analysis: contextarena.ai/variant-bias…
@Pokee_AI
3/3
Context Arena: Added @Alibaba_Qwen's Qwen 3.8 Max on 8-needle GDM-MRCRv2!
At xhigh thinking, it goes toe-to-toe with frontier models like Claude Opus 5 and GPT-5.6 Sol up to 128k, hitting a massive 96.1% AUC@128k with >92% accuracy across the board. But push past 128k, and scores take a steep dive.
All results at: contextarena.ai
Highlights:
- Scores 96.1% AUC@128k (96.9% average) at xhigh reasoning, ahead of medium (93.3%) and low (91.0%).
- Stays above 92% accuracy across all tests up to 128k context (98.7% at 8k, 97.5% at 16k, 99.3% at 32k, 96.5% at 64k, 92.3% at 128k).
- Extra thinking time pays off past 32k: xhigh pulls ahead at 64k (96.5% vs 93.3%) and 128k (92.3% vs 87.3% on medium, 81.5% on low).
- Drops off at extreme lengths: past 128k, scores fall across all tiers to ~68–71% at 256k and ~26–30% at 512k.
@ 128k (AUC / cumulative average):
- Claude Opus 5 (max): 97.5% / 97.8%
- GPT-5.6 Sol (max): 97.1% / 97.9%
- Qwen 3.8 Max (xhigh): 96.1% / 96.9%
- Gemini 3.7 Flash (high): 93.9% / 96.0%
- Qwen 3.8 Max (medium): 93.3% / 95.2%
- Qwen 3.8 Max (low): 91.0% / 93.1%
- GPT-5.6 Terra (max): 89.4% / 92.6%
- Gemini 3.6 Flash (high): 84.2% / 89.3%
- DeepSeek V4 Pro 0813 (high): 81.1% / 86.9%
@ 512k:
- Gemini 3.7 Flash (high): 77.7%
- GPT-5.6 Sol (max): 61.9%
- Gemini 3.6 Flash (high): 61.1%
- GPT-5.6 Terra (max): 49.8%
- Claude Opus 5 (max): 42.5%
- DeepSeek V4 Pro 0813 (high): 33.4%
- Qwen 3.8 Max (xhigh): 29.7%
- Qwen 3.8 Max (low): 28.5%
- Qwen 3.8 Max (medium): 25.7%
These models, side by side: contextarena.ai/?needles=8&s…
Enjoy!
@Alibaba_Qwen @AnthropicAI @OpenAI @GoogleDeepMind @deepseek_ai
1/3
Reasoning tiers up to 128k, bin by bin (xhigh / medium / low):
- 8k: 98.7% / 99.3% / 99.3%
- 16k: 97.5% / 97.9% / 96.7%
- 32k: 99.3% / 98.1% / 94.8%
- 64k: 96.5% / 93.3% / 93.3%
- 128k: 92.3% / 87.3% / 81.5%
- 256k: 67.7% / 70.6% / 71.2%
- 512k: 29.7% / 25.7% / 28.5%
Thinking tokens & cost through 128k:
- xhigh: 96.1% AUC@128k, ~5,800 thinking tokens/sample at 128k (3,318 avg across run), ~$64.44 estimated standard cost through 128k
- medium: 93.3% AUC@128k, ~500 thinking tokens/sample at 128k (1,286 avg across run), ~$52.21 estimated standard cost through 128k
- low: 91.0% AUC@128k, ~380 thinking tokens/sample at 128k (888 avg across run), ~$51.98 estimated standard cost through 128k
Takeaways:
- Up to 32k context, medium thinking is practically as good as xhigh (98.1% vs 99.3%) for much less cost and latency.
- At 64k and 128k, maxing out thinking pays off clearly: xhigh gains +5.0 points over medium and +10.8 points over low at 128k.
- Past 128k, more thinking no longer helps: at 256k and 512k, low and medium match or slightly edge out xhigh, with all tiers dropping to 25–30% at 512k.
Notes:
- Reliability: 861/861 tests completed with 0 errors on xhigh; low had 1 error and medium had 2 errors.
@Alibaba_Qwen
2/3
Looking at mistake patterns across all 861 outputs on xhigh: 75.1% of errors picked another valid version of the requested needle rather than hallucinating or picking an unrelated needle. Output formatting was near-flawless with 99.9% hash prefix adherence.
Position takeaways:
- Asking for the 1st needle is easiest: target #1 was found 94.9% of the time, scoring ~23 points higher than targets #2 through #8 (~66–82%).
- Earlier in the context is easier: targets placed in the first 20% of the document scored ~25 points higher than targets placed in the final 20% (89.5% vs 65.0%).
- No boost for the end: there was no noticeable bump for the final needle (#8) or document edges.
Across tiers:
- Medium and low show very similar primacy effects (~24–25 points higher on needle #1, and ~18–22 points drop towards the end of context).
Full analysis: contextarena.ai/variant-bias…
@Alibaba_Qwen
3/3
Context Arena: Quick update for several models added to 8-needle GDM-MRCRv2 recently (Aug 11th+) without a proper post about them. This summarizes the remaining ones.
Full leaderboard: contextarena.ai
@ 128k (AUC / cumulative average):
- Claude Opus 5 (max): 97.5% / 97.8%
- Ling 3.0 Flash (enabled): 66.9% / 74.8%
- Muse Spark 1.1 (medium): 66.8% / 74.6%
- Muse Spark 1.1 (low): 64.1% / 71.1%
- Kimi K3 (max): 64.0% / 73.3%
- Muse Spark 1.1 (minimal): 63.1% / 70.9%
- Nex-N2-Mini (reasoning off): 60.3% / 68.7%
- Muse Spark 1.1 (high): 60.2% / 70.0%
- Muse Spark 1.1 (xhigh): 59.4% / 69.7%
- Nex-N2-Mini (enabled): 59.2% / 67.3%
- Kimi K3 (high): 58.5% / 67.6%
- Hy3 (high): 52.8% / 68.3%
- Hy3 (low): 51.7% / 67.6%
- Seed 2.1 Turbo (enabled): 50.4% / 63.3%
- Kimi K3 (low): 49.4% / 60.0%
- Solar Pro 4 (enabled): 48.0% / 61.2%
- Ling 3.0 Flash (reasoning off): 44.3% / 51.2%
- Seed 2.1 Turbo (reasoning off): 27.5% / 36.2%
- Inkling Small (max): 24.9% / 37.8%
- Solar Pro 4 (reasoning off): 24.3% / 28.9%
- Inkling Small (high): 21.9% / 34.0%
- Inkling Small (medium): 20.8% / 30.9%
- Inkling Small (low): 20.5% / 28.9%
- Inkling Small (minimal): 20.0% / 29.7%
- Hy3 (reasoning off): 20.0% / 26.9%
- Inkling Small (reasoning off): 14.1% / 16.6%
- Laguna S 2.1 (enabled): 11.6% / 15.1%
- Laguna XS 2.1 (enabled): 9.5% / 14.7%
Enjoy!
@AnthropicAI @InclusionAI @Kimi_Moonshot @ByteDanceSeed_ @upstageai @TencentHunyuan @thinkymachines @poolsideai
With frontier models starting to saturate 8-needle MRCRv2 at ≤128k context, I'm working on a new 16-needle benchmark with new testing combinations.
16-needle entry criteria:
- ≥75% pointwise at 128k on 8-needle
- Highest reasoning tier only (for budget reasons; when 8-needle retires I'll add more reasoning tiers)
Context Arena: Added @xai's Grok 4.6 on 8-needle GDM-MRCRv2!
All results at: contextarena.ai
Highlights:
- 74.4% AUC@128k at medium, gaining +32.1 points over Grok 4.20 (42.3%), with non-overlapping 95% CIs across all bins from 16k through 256k.
- Peer standing through 256k: at 74.4% AUC@128k and 46.8% at 256k, medium matches GPT-5.6 Luna (max, 74.8% / 45.1%) and Qwen 3.7 Flash (74.2% / 43.1%).
@ 128k (AUC / cumulative average):
- GPT-5.6 Luna (max): 74.8% / 82.1%
- Muse Spark 1.2 (high): 74.6% / 81.8%
- Grok 4.6 (medium): 74.4% / 82.0%
- Qwen 3.7 Flash (enabled): 74.2% / 81.2%
- Grok 4.6 (xhigh): 73.4% / 81.5%
- Grok 4.6 (high): 71.9% / 79.4%
- Claude Sonnet 5 (max): 71.6% / 80.0%
- Qwen 3.6 Plus (enabled): 70.9% / 79.7%
- DeepSeek V4 Flash 0731 (max): 70.4% / 77.9%
- Grok 4.6 (low): 63.6% / 75.0%
- Grok 4.20 (enabled): 42.3% / 55.1%
- Grok 4.20 (reasoning off): 15.9% / 21.0%
@ 256k (pointwise):
- Muse Spark 1.2 (high): 59.0%
- Claude Sonnet 5 (max): 52.2%
- Grok 4.6 (xhigh): 47.7%
- Grok 4.6 (medium): 46.8%
- Grok 4.6 (high): 45.8%
- GPT-5.6 Luna (max): 45.1%
- DeepSeek V4 Flash 0731 (max): 44.5%
- Qwen 3.7 Flash (enabled): 43.1%
- Grok 4.6 (low): 41.8%
- Grok 4.20 (enabled): 15.9%
- Grok 4.20 (reasoning off): 10.4%
These models, side by side: contextarena.ai/?needles=8&s…
Enjoy!
@xai @OpenAI @AnthropicAI
1/3
Grok 4.6 (medium) vs Grok 4.20 (enabled), pointwise:
- 8k: 98.3% / 95.2%
- 16k: 96.9% / 59.9%
- 32k: 91.5% / 61.9%
- 64k: 77.0% / 39.0%
- 128k: 46.2% / 19.4%
- 256k: 46.8% / 15.9%
We didn't bench Grok 4.5 or 4.3, so for these results we are comparing against Grok 4.20.
The jump is consistent across context sizes. Every bin from 16k through 256k shows non-overlapping 95% confidence intervals: +37.0 points at 16k, +29.6 at 32k, +38.0 at 64k, +26.8 at 128k, and +30.9 at 256k.
Reasoning tiers through 128k:
- medium: 74.4% AUC@128k, 2,945 reasoning tokens/sample
- xhigh: 73.4% AUC@128k, 3,649 reasoning tokens/sample
- high: 71.9% AUC@128k, 3,193 reasoning tokens/sample
- low: 63.6% AUC@128k, 882 reasoning tokens/sample
@xai @OpenAI @AnthropicAI
2/3
Variant-bias analysis checked all 625 outputs from Grok 4.6 (medium): 456 correct, 161 wrong variant, 5 wrong needle, and 3 unmatched. 95.3% of errors (161/169) selected another valid version of the requested needle rather than an unrelated one, and the required hash prefix was present in 99.8% of responses.
Position findings:
- Asking for the first needle is easier: target 1 was correct 96.6% of the time, 26.8 points higher than targets 2–8 (95% CI: 6.8 to 41.4; q=0.002).
- Needles near the start score higher: targets in the first 20% of the text scored 34.4 points above the last 20% (93.3% vs 58.1% correct; 95% CI: 10.6 to 47.8; q=0.002).
There is no statistically supported boost for the final needle (target 8 vs 1–7: -9.6 points, q=0.085) or document edges (outer 40% vs middle 60%: +3.8 points, q=0.739). Target order and text position align closely in this dataset, so these two effects reflect related patterns.
Across reasoning tiers, medium, high, and xhigh show very similar position curves (target 1 at 96.6% to 97.7%, first-quintile at 90.8% to 94.1%). On low, the drop-off is sharper: first-part accuracy remains at 95.0%, but drops to 52.4% in the final 20% of the text.
Full analysis: contextarena.ai/variant-bias…
@xai
3/3
