Joan Gamell retweeted
Today, we're announcing our Series B: $200M at a $3.1B valuation, and the release of Arena's Alignment Index.
AI is advancing faster than our ability to evaluate it. The world needs a neutral third party to measure how safe and aligned AI actually is once it's in the hands of real people. That's the role Arena is stepping into today.
The Alignment Index is built from real-world agent traces, starting with three signals: Unauthorized Action, False Attribution, and Deceptive Completion, with results for over 20 frontier models launching today.
Hear more from our CEO @ml_angelopoulos below, and read the full breakdown, and more details about our company growth in thread.
This funding will allow us to go further on safety signals, agentic capabilities, and modalities. It will also allow us to grow the company to match the scale of our mission. Today, we’re still a nimble team of 90, and we're looking for people to join us across research, product, engineering, and more. Come build with us.
As AI gets more capable and autonomous, understanding how well it can safely and effectively work for people only matters more. That's what Arena is built to measure and advance.
Thank you to our community, our investors, and everyone building with us!
Take a look at our open roles if you want to join us (link in bio).
Introducing the Arena Alignment Index, our new benchmark measuring safety and alignment of AI agents in real-world use.
Built from 90K+ real-world agent sessions across 27 models, the index measures three critical signals:
- Unauthorized Action (UA): Taking actions beyond the user's instructions or permissions
- False Attribution (FA): Attributing statements or actions that are contradicted by user-provided evidence
- Deceptive Completion (DC): Claiming a task was completed when it was not.
Key findings:
- OpenAI models currently lead the Alignment Index
- Rogue actions are rare, but can have serious consequences when they occur
- Agents can mislead users about task progress
- Misalignment risks increase with conversation length
- Safety and alignment are improving across model generations
As shown in the leaderboard below (sorted by lab), @OpenAI’s GPT-6.1-Sol leads the Arena Alignment Index with a score of 87.9, followed by @AnthropicAI’s Claude-Opus-5.5 at 83.2 and @spacexai's Grok-4.7 at 82.7. OpenAI also has the best observed rates across all three signals: 0.89% Unauthorized Action, 1.98% False Attribution, and 2.34% Deceptive Completion.
Across all four labs, newer models consistently outperform their predecessors, suggesting broad progress in agent safety and alignment. As agents take on longer, more complex, and higher-stakes tasks, measuring not just what they can accomplish, but how safely and reliably they act, becomes increasingly important.
This marks an important step toward making safety and alignment a core part of how Arena evaluates AI. The index is an initial starting point, and we'll continue expanding the index with additional safety signals and models over time.
More analysis below👇
Joan Gamell retweeted
Do models need their native harness for coding?
Awesome work from our intern @melissapan on this. She dug into whether the harness (Claude Code vs Codex CLI vs Pi) actually moves the needle for coding agents.
Turns out it matters way less than people assume. 21 model-harness pairs, real rigor. More from Arena to come.
More details on the Arena blog: arena.ai/blog/coding-agents-…
Does your Claude model really need Claude Code…? 🤔
We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:
1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.
Millions of people are using coding agents, but the impact of harness choice remains unclear.
(1/n) More details in the thread. 🧵
Joan Gamell retweeted
We analyzed how @claudeai's writing has changed from Fable 5 to Fable 5.1 across tens of thousands of high-reasoning Text Arena outputs.
Overall, Fable 5.1 by @AnthropicAI uses fewer agreement openers, fewer em dashes and less wording like “honestly” and “frankly”, while its answers have grown longer.
Compared with Fable 5:
– Agreement openers such as “yes” and “exactly” are 58% less common, appearing in 0.99% of responses versus 2.35%.
– Honesty wording such as “honestly” and “frankly” falls 45% per 1,000 words.
– Em dashes fall 32% per 1,000 words.
It has also shifted some patterns in another direction, e.g.: semicolons increase 63% per 1,000 words.
Fable 5.1 still writes long answers, with sentence length and clause frequency close to Opus 5:
– Median response length rises 30% from Fable 5: 319 → 414 words.
– That remains 21% shorter than Opus 5’s 525 words.
– Its sentence-length measure is slightly higher than Opus 5’s: 12.54 versus 12.20 words per sentence.
Other familiar patterns also become less common compared with Fable 5:
– Stock phrases such as “load-bearing” fall 20% per 1,000 words.
– Hedges and caveats such as “perhaps” and “arguably” fall 36% per 1,000 words.
– Praise and validation appear in 1.98% of responses, down from 3.17%.
Long content words fall from 42.6% to 38.6%, while abstract nouns fall 25%, from 4.39 to 3.28 per 100 words.
Let us know what you think in the comments.
We analyzed both @claudeai Fable 5 and Fable 5.1 on writing clarity ranked by humans, and are sharing results later today.
Which model do you think writes more clearly?
20%Fable 5
80%Fable 5.1
1,224 votes • Final results Joan Gamell retweeted
We analyzed tens of thousands of real-world, high-reasoning Claude Opus and Fable outputs from versions 4.5 to 5 in the Text Arena and found that @claudeai's writing has changed in more ways than one:
Across language complexity, patterns and markers, we now see:
Claude’s answers have become much longer:
- Opus 5 averages 510 words per response
- This makes Opus 5 responses 3x longer than Opus 4.5’s average of 158
Responses have also become more structurally elaborate from Opus 4.5 to Opus 5:
- 58% rise in average sentence length
- 46% rise in clause frequency rises
However, the vocabulary itself is not becoming more difficult, making Opus 5 longer and more structurally complex, but lexically simpler between 4.5 and 5:
- 6.8% pt drop in long content words (46.9% to 40.1%)
- 53% drop in abstract nouns (6.02 → 3.79 per 1k words)
Writing habits people notice are much more common in Opus 5:
- 2.3x as many em dashes
- ~2x as many phrases like “load-bearing”
- 50% more honesty wording like “honestly” and “frankly”
Fable 5 is 38% more concise than Opus 5, averaging 316 words vs. 510. At the same time, it is nearly 2x as likely to include praise/validation or open with phrases like “yes, exactly”
Joan Gamell retweeted
agent mode on @arena is really good, if you haven't used it you should
somehow it regularly blows the frontier labs out of the water because the experiences are so chatmaxxed they just shit themselves on even simple actions
Check out this quick retrieve data -> plot it query:
Joan Gamell retweeted
Exciting news: Claude Fable 5 ranks #1 on the new Agent Arena leaderboard!
Fable 5 leads by the widest margin ever over Opus-4.8 and GPT-5.5 on two key signals: confirmed task success rate and praise vs. complaint, despite weaker steerability. If Fable can do something, it will do it very well. If it can't/doesn't want to do something, it may be hard to steer the model towards the goal.
In Agent Arena, we measure models on millions of real-world, long-horizon agentic tasks. Models get web search, filesystem, and terminal tools to complete complex workflows: writing code, creating slide deck, researching the web, building apps, and analyzing documents.
We use the causal tracing methodology to measure a model's net improvement which indicates how much it improves outcomes relative to the average model.
Huge congrats to @AnthropicAI for the incredible milestone! Below we break down how Claude Fable 5 (based on Mythos) scored across 5 signals, drawn from tasks submitted by a global community of users.
Introducing Agent Arena: real-world agentic evals at scale.
How do you evaluate agents doing actual work? We measure millions of live sessions where real users accomplish real tasks.
On Arena, models now get web search, filesystem, and terminal tools to complete complex workflows: writing code, creating slide deck, researching the web, building apps, and analyzing documents.
Every session produces rich signals. Users iterate with the agent turn-by-turn: approving, editing, correcting, praise or expressing frustration. The environment gives feedback too: shell errors, tool failures, recovery attempts, and more.
Our leaderboard measures each model's agentic performance using causal inference across five signals: task success, steerability, error recovery, user praise vs. complaint, and tool hallucination.
This leaderboard snapshot is built from 300K+ tasks, 2M+ tool calls, and 40M lines of code by agents.
Top labs in Agent Arena:
- #1 @OpenAI: GPT-5.5 (High)
- #2 @AnthropicAI: Claude-Opus-4.7 (Thinking)
- #3 @Zai_org: GLM-5.1
- #4 @GoogleDeepMind: Gemini-3.1-Pro
- #5 @Kimi_Moonshot: Kimi-K2.6
More analysis in the thread, with the full technical blog below.
Joan Gamell retweeted
Introducing Agent Arena: real-world agentic evals at scale.
How do you evaluate agents doing actual work? We measure millions of live sessions where real users accomplish real tasks.
On Arena, models now get web search, filesystem, and terminal tools to complete complex workflows: writing code, creating slide deck, researching the web, building apps, and analyzing documents.
Every session produces rich signals. Users iterate with the agent turn-by-turn: approving, editing, correcting, praise or expressing frustration. The environment gives feedback too: shell errors, tool failures, recovery attempts, and more.
Our leaderboard measures each model's agentic performance using causal inference across five signals: task success, steerability, error recovery, user praise vs. complaint, and tool hallucination.
This leaderboard snapshot is built from 300K+ tasks, 2M+ tool calls, and 40M lines of code by agents.
Top labs in Agent Arena:
- #1 @OpenAI: GPT-5.5 (High)
- #2 @AnthropicAI: Claude-Opus-4.7 (Thinking)
- #3 @Zai_org: GLM-5.1
- #4 @GoogleDeepMind: Gemini-3.1-Pro
- #5 @Kimi_Moonshot: Kimi-K2.6
More analysis in the thread, with the full technical blog below.
Introducing Agent Mode: Agentic AI is now measured in the Arena.
Agent Mode can do deep research, create reports, generate images, build websites, debug code, and more.
It completes more complex tasks by using tools like web search, bash in a sandbox environment, image generation, file writing, and asking follow-up questions.
Frontier models are waiting for you in Agent Mode to take on real-world tasks. GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and top open models. Test them yourself.
"Build an Agent testing framework, make no mistakes"
Not as easy as it sounds! Congrats team!!
Introducing Agent Mode: Agentic AI is now measured in the Arena.
Agent Mode can do deep research, create reports, generate images, build websites, debug code, and more.
It completes more complex tasks by using tools like web search, bash in a sandbox environment, image generation, file writing, and asking follow-up questions.
Frontier models are waiting for you in Agent Mode to take on real-world tasks. GPT-5.5, Claude Opus 4.7, Gemini 3.1 Pro, and top open models. Test them yourself.
Joan Gamell retweeted
A brief history of me
ALT Bell curve meme showing that both the "low IQ" and "high IQ" sides just say, “Just use Cursor,” while the midwit in the middle overcomplicates everything with Claude Code, LSP, subagents, RAG, Context7, etc.
Joan Gamell retweeted
It's true. Here's a plot of GPT models and their usage of "goblin", "gremlin", "troll", etc over time. There's no anti-gremlin system instruction on our side, we get to see GPT-5.5 run free.
Joan Gamell retweeted
Exciting news - DeepSeek V4 Pro is in the Arena with 1.6T parameters (49B activated) alongside V4 Flash at 284B parameters (13B activated). Both support 1M token context. It’s a major leap over DeepSeek V3.2!
Code Arena:
- DeepSeek V4 Pro (thinking): #3 open model (#14 overall), on par with GPT-5.4-high and Gemini-3.1-Pro in agentic webdev tasks
Text Arena:
- DeepSeek V4 Pro (thinking): #2 open model (#14 overall), matching Kimi-2.6
- DeepSeek V4 Flash (thinking): #10 open model (#47 overall)
Competition at the top of the open model leaderboards keeps heating up. Huge congrats to @DeepSeek_AI on the strong comeback!
🚀 DeepSeek-V4 Preview is officially live & open-sourced! Welcome to the era of cost-effective 1M context length.
🔹 DeepSeek-V4-Pro: 1.6T total / 49B active params. Performance rivaling the world's top closed-source models.
🔹 DeepSeek-V4-Flash: 284B total / 13B active params. Your fast, efficient, and economical choice.
Try it now at chat.deepseek.com via Expert Mode / Instant Mode. API is updated & available today!
📄 Tech Report: huggingface.co/deepseek-ai/D…
🤗 Open Weights: huggingface.co/collections/d…
1/n
Joan Gamell retweeted
Exciting news - GPT-Image-2 by @OpenAI has claimed the #1 spot across all Image Arena leaderboards!
A clean sweep with a record-breaking +242 point lead in Text-to-Image - the largest gap we’ve seen to date.
- #1 Text-to-Image (1512), +242 over #2 (Nano-banana-2 with web-search aka gemini-3.1-flash-image)
- #1 Single-Image Edit (1513), +125 over #2 (Nano-banana-pro aka gemini-3-pro-image)
- #1 Multi-Image Edit (1464), +90 over #2 (Nano-banana-2)
No model has dominated Image Arena with margins this wide.
Huge congratulations to @OpenAI on this major breakthrough in image generation! More performance breakdowns by category in the thread below.
Joan Gamell retweeted
We’ve added Pareto frontier charts to the leaderboard.
Now available across:
Text, Vision, Search, Document, and Code Arena.
The Pareto frontier curve demonstrates which models are most efficient at their level of performance (by Arena score) vs. a blended price per 1M tokens (3:1 Ratio).
Text Models on the Pareto Frontier curve today:
- Claude Opus 4.6
- Gemini 3.1 Pro Preview
- Grok 4.20 Beta Reasoning
- Gemini 3 Flash
- Grok 4.1 Thinking
- DeepSeek v3.2 Exp Thinking
- Deepseek v3.2
- Mimo v2 Flash (non-thinking)
- Gemma-3-27b-it
- Gemma-3-12b-it
- GPT-OSS-20b
- Gemma 3n-e4b-it
Model count by Lab for Text Arena:
- 5 models @GoogleDeepMind
- 2 models from @xai
- 2 models from @deepseek_ai
- 1 @AnthropicAI
- 1 @Xiaomi
- 1 @OpenAI
Dig into the Pareto frontier for Document, Search, Video or Code Arena, by lab, or the expert categories most relevant to you.
Joan Gamell retweeted
Did you know? We’re funding independent research in AI evaluation and measurement—up to $50k per project.
The Q1 deadline to apply for Arena’s Academic Partnerships Program is March 31.
AI needs better evaluations.
Today we’re announcing Arena’s Academic Partnerships Program to fund independent academic research in AI evaluation and measurement.
▫️Up to $50K/project. Q1 Deadline: March 31, 2026.
See more in thread for details and how to apply 👇
Joan Gamell retweeted
Today we’re launching the Video Edit Arena to evaluate the frontier capability of video models!
- #1 Grok-Imagine-Video, @xAI
- #2 Kling-o3-pro, @Kling_ai
- #3 Kling-o1-pro, @Kling_ai
- #4 Gen4-aleph, @Runwayml
The leaderboard is powered by thousands of real-world community votes. Click the Edit button in Video Arena to edit any video and compare top model outputs.
More models coming soon!
Joan Gamell retweeted
Today, we’re launching a dedicated Multi-File React leaderboard. When Code Arena first launched, we evaluated models on single-file HTML.
Then we raised the bar → multi-file React apps (routing, hooks, components, state management) and now have a leaderboard to match!
Single-file HTML tests instruction following.
Multi-file React tests more complex agentic capabilities:
• Cross-file coordination
• Component architecture
• Dependency management
• State management
• Build reliability
Now see how models stack up under both scenarios. Learn more in thread 🧵👇
Multi-file apps are now live in Code Arena!
Since launching Code Arena in November to evaluate frontier AI models on real-world, agentic coding tasks, we’ve received a lot of feedback asking to adapt more complex workflows.
With multi-file apps, you can now build and compare production-ready projects, making it easier to evaluate how top frontier AI models perform on your actual use cases.
Joan Gamell retweeted
⚡️Who powers the Arena leaderboard? You do.
But not all votes are Arena level research-grade quality.
Every score is built from real-world prompts and human input, continuously refreshed as the way we use AI evolves.
In this video, ML Scientist @cthorrez explains how votes are processed and validated into high-quality data trusted by leading AI labs. Dive into:
▪️ Filtering low-signal and bot-like activity
▪️ Our quality control and validation pipeline
▪️ Ensuring all votes are based on blind pairings
▪️ How diverse user prompts differ from static benchmarks
Arena scores reflect how models perform in the wild: across coding, creative work, expert tasks, and everyday use.
Watch the breakdown to see how your vote helps shape the frontier.
Joan Gamell retweeted
✨NEW: Arena Leaderboard UI Updates
Millions of votes power the leaderboard. Now you can filter for what matters to you.
A new side panel lets you filter and break down ranked results to find the best model for your task. Some highlights:
• Filter by category (e.g. Coding, Expert prompts)
• Open vs. Proprietary Models
• Rank labs by their top-performing models
…just to name a few. Check it out, and let us know what you think.
Joan Gamell retweeted
The new @xAI Grok-Imagine-Image model is a Pareto-optimal model in Image Arena:
The Pareto frontier tells us which model has the highest Arena score at each price point. @xAi’s latest models have improved the frontier, giving optimal performance in the mid-price tier. For a wide range of prices between 2c and 8c per image, @elonmusk’s @xAI has the leading model, delivering the maximum performance.
Top models on the Pareto frontier for Image Arena (Single Image Edit):
- @OpenAI: GPT-Image-1.5-high-fidelity
- @xAI: Grok Imagine Image Pro
- @xAI: Grok Imagine Image
- @bfl_ml: Flux 2 Klein 9B
- @bfl_ml: Flux-2-Dev
- @reve : V1.1 Fast
See thread for how the frontier changes for Text-to-Image 🧵
Latest image models from @xAI, Grok-Imagine-Image and Pro debut top 6 in the Image Arena!
Text-to-Image:
▪️ #4 Grok-Imagine-Image; scoring 1170, surpassing Flux-2-max and Nano-banana
▪️ #6 Grok-Imagine-Image-Pro
Image-Edit:
▪️ #5 Grok-Imagine-Image-Pro; scoring 1330, overtaking Seedream-4.5
▪️ #6 Grok-Imagine-Image
With this launch, @xAI is now a top-3 Image AI provider alongside @GoogleDeepMind and @OpenAI.
Congrats to the @xAI team on the impressive releases!
