Co-Founder @ArtificialAnlys | Message me to play 🎾 in SF
San Francisco
Joined January 2022
- Tweets316
- Following393
- Followers1.1K
- Likes896
Regarding neocloud financials, people don’t realize that if there are chip shortages and limits on datacenters being built then this can be good for many of the debt holders while simultaneously bad for equity holders.
Debt holders need to make sure they get their money back and high prices support this while equity holders want growth
George Cameron retweeted
Welcome to the era of embodied AI.
We are launching AA-Robotics v0.1, our research preview that measures frontier language model performance on understanding, planning, and acting in the physical world. In v0.1, each model is given control of a 6-axis robot arm (5 joints and a gripper) and a simple objective. We’re releasing results for our first eight tasks today.
We find that frontier language models in coding agent harnesses have recently become capable of directly manipulating robot arms for simple tasks. Today, they are far too slow and expensive for any kind of production usage - nearly two orders of magnitude worse than human baselines. We expect this to change quickly.
GPT-6 Astra places first in AA-Robotics v0.1, followed by GPT-6.1 Sol, Gemini 4 Argon and Claude Opus 5.5.
AA-Robotics v0.1 details:
➤ Our first eight tasks test a range of capabilities: control and dexterity (precise grasping and object handling), spatial reasoning (how objects interact with one another) and adaptability (how models respond to the unexpected)
➤ Models get only brief instructions: how to connect to the robot arm (via a simple HTTP API), the arm's name and how to submit an attempt. They can view the scene through four cameras and write and run code to plan and send movements
➤ Once a model submits an attempt, its end state is graded against the task's pass criteria. A pass scores 100; an incomplete attempt earns partial credit. All attempts are zero-shot
Key findings:
➤ The GPT family tops the leaderboard: GPT-6 Astra scores 69, ahead of GPT-6.1 Sol at 65 and Gemini 4 Argon at 55. Claude Opus 5.5 (49) and Claude Fable 5.1 (46) trail
➤ Today’s models are incredibly slow at our tasks, taking ~30 minutes on average to complete each tabletop task
➤ Cost varies widely: GPT-6.1 Sol is the cheapest at $1.93 per task on average; GPT-6 Astra, the most expensive, costs $16.36, more than eight times as much
➤ Compared with a human, today's best-performing frontier language models are ~100x slower and ~10x more expensive
More results, clips, and how AA-Robotics v0.1 works follow in the thread below.
George Cameron retweeted
Announcing Artificial Analysis for Government: benchmarking, advisory and data to help public sector clients design AI policy and select AI technologies
Our services help public sector clients to:
➤ Design AI policy and support decision-making: Our benchmarking and expertise can inform the urgent decisions governments must make in navigating AI - for example, how to secure critical infrastructure, safeguard employment and growth, and balance national priorities including innovation, competition, and security. We recently signed an MOU with the Korean Government as an independent evaluator supporting Korea’s efforts to develop globally competitive frontier AI models and strengthen its domestic AI ecosystem.
➤ Select AI Technologies: We are the go-to resource for millions of users navigating AI. We can support governments to deploy the most effective agents at the lowest cost, empower public sector workers, and plan for future use cases.
Our topic expertise includes:
➤ Capability, cost and performance: We are the global standard in measuring capabilities, accuracy, hallucination, cost, speed and latency of models and agents. Our benchmarking is widely referenced by top AI labs and news outlets.
➤ Cybersecurity and safety: The Artificial Analysis Cyber Index Alliance brings together partners including NVIDIA, IBM, Vercel and Collinear to set a new standard for evaluating AI cyber defense. This launched alongside the Artificial Analysis Cyber Index, which evaluates how well agents find and fix vulnerabilities. On safety, we evaluate models on a broad spectrum of harms across escalating levels of adversarial pressure. We recently partnered with NVIDIA to benchmark guardrail models for safety and refusal.
Get in touch at the link below
George Cameron retweeted
Introducing the Artificial Analysis Decision Index, benchmarking decision models across classification, tool calling, knowledge and reasoning. Jev leads among decision models; language models such as Claude Haiku 5.5 still deliver higher accuracy, but at much higher cost and latency
Decision models deliver structured answers or classifications with associated probabilities, offering faster responses and lower costs than using language models. The Artificial Analysis Decision Index v1.0 compares the capability, cost, and speed of decision models and also compares to language models.
Benchmarks in the Artificial Analysis Decision Index v1.0:
➤ AA-Omniscience-Decision: A private dataset of 500 questions adapted from the proprietary AA-Omniscience benchmark, measuring decision models’ knowledge and tendency to hallucinate
➤ HLE-MultiChoice: 513 text-only multiple-choice questions from Humanity’s Last Exam, testing decision models’ reasoning capabilities
➤ When2Call: A tool-calling benchmark from NVIDIA, testing whether models can correctly decide to call a tool, request clarification, decline to answer, or respond directly, evaluating capabilities relevant to agentic workflows and tool routing
➤ WildGuardTest: A safety benchmark evaluating models’ classification capabilities across harmful user requests, assistant refusals, and harmful responses. We use a subset of 835 of the original 1,725 test rows where all three human annotators agreed on all three labels, after excluding questions that consistently trigger model refusals
Key results:
➤ Jev 1.13 leads the Decision Index, and sits on the Capability vs. Cost Pareto frontier: Jev 1.13 from @typesafeai scores 58 on the Artificial Analysis Decision Index, ahead of GLiDE (54) from @fastinoAI and Microsoft-Decision-1 (54) from @Microsoft. Decider v1.1 27B from @perplexity_ai has the lowest Cost per 1M Tasks at $17, with Microsoft-Decision-1 ($31) and Jev 1.13 ($42) also on the Capability vs. Cost Pareto frontier. Celeris-1 Decision has the fastest Time per Task at 0.06 seconds, more than 2x faster than Jev 1.13 (0.13 seconds per task)
➤ Decision models are poorly calibrated on knowledge and reasoning tasks: Across almost all probability levels for chosen answers, accuracy falls below that probability for all decision models. This gap is largest on AA-Omniscience-Decision and HLE-MultiChoice, while most models are relatively well-calibrated on WildGuardTest (classification) and When2Call (tool calling)
➤ Decision models offer significant speed and cost advantages over language models, but trail on knowledge and reasoning: On classification and tool-calling tasks, leading decision models perform comparably to language models while being significantly faster and cheaper. However, language models substantially outperform on knowledge and reasoning. Claude Haiku 5.5 (Low) scores 61 on the Decision Index, 3 points ahead of Jev 1.13, but takes 1.4 seconds per task vs. 0.13 seconds (11x slower) and costs $184 per 1M Tasks vs. $42 for Jev 1.13 (4.4x more expensive)
We will continue to develop the Artificial Analysis Decision Index to help developers and companies better understand when and how to use decision models.
George Cameron retweeted
“Do you think that getting married young has made you less ambitious?”
Welcome to the first episode of SOTA Wives with Sarah Chieng (Head of DevX at Cerebras, @MilksandMatcha) x Vivian Midha Shen (General Partner at Y Combinator, @vivianmshen).
In episode 1: finding your person before getting rich, turning down startup funding to finish MIT, what a YC partner sees in founders, and how a birthday post became “psychological violence.”
4:08 Finding your husband before the exit
8:48 Did we marry away our ambition?
18:04 The internet called Sarah's birthday “psychological violence”
20:47 Does your YC partner secretly think you’re doomed?
23:47 Building a startup with your ex-boyfriend
28:54 SF cat parties
WTF is a SOTA Wife?
SOTA Wives is a podcast exploring the unfiltered side of Silicon Valley.
We built it as an antidote to the performative polish of tech Twitter.
Brought to you by @MilksandMatcha x @vivianmshen.
Episode 1 goes live on Monday, 12th October at 9 AM PT on everywhere you get your podcasts 💙
George Cameron retweeted
We have measured intelligence. Now it's time to see it.
We are launching AA-3D, our new benchmark for how well AI models build interactive 3D scenes. Each model works as an agent in an offline sandbox and turns a written brief into a three.js web page. A human panel of judges compares pairs of scenes built from the same brief without knowing which model made which.
AA-3D details:
➤ 55 written briefs across seven domains, including landmarks, machines, living bodies, games, and space. Each brief says what the scene must show and how people should interact with it, and leaves the rest to the model
➤ Each model gets up to three hours and 250 turns. Its sandbox has Blender, Rapier physics, Python, a headless browser to test its own work, and a CC0 library of materials, 3D models, and characters
➤ Panelists spend at least 30 seconds with each scene before they vote. One comparison in every four is a validation check
➤ Over 10,000 counted votes from over 1,000 panelists rank 21 models by Elo, with 95% confidence intervals
Key findings:
➤ Four models are statistically tied at the top: Claude Sonnet 5.5 (1194), Claude Opus 5.5 (1190), GPT-6 Astra (1165), and Claude Fable 5.1 (1131). The 95% confidence intervals of the other three overlap Claude Sonnet 5.5's
➤ AA-3D Elo rises with intelligence: each point on the Artificial Analysis Intelligence Index is worth about 18 Elo (Pearson 0.92)
➤ The best-placed open weights models, Kimi K3 (982) and MiMo-V2.6-Pro (977), rank 11th and 12th, more than 200 points behind first place
➤ A scene costs $0.20 to $32.08, depending on the model. The cheapest, GPT-6 Luna, ranks eighth
More results, more scenes, and how AA-3D works follow in the thread below.
George Cameron retweeted
Can today's frontier language models control a real robot arm, zero-shot, with no coaching?
AA-Robotics v0.1 coming soon.
George Cameron retweeted
Google’s new Gemini 4 Argon equals GPT-6 Astra on the Artificial Analysis Intelligence Index at 60% of the Cost per Task with discounted prices. Google is now back to being one of the top three labs in intelligence achieved
Gemini 4 Argon is @GoogleDeepMind’s first proprietary model above the Flash class in over 7 months. With high reasoning (the highest available), it scores 53 on the Artificial Analysis Intelligence Index, matching GPT-6 Astra (max, 53) and 1 point ahead of GPT-6.1 Sol (max, 52), with gains driven by lower hallucinations and stronger agentic capabilities.
At its current 50% pricing discount and with cache discounts increased to 95%, Gemini 4 Argon costs $1.99 per Intelligence Index task, 60% of GPT-6 Astra (max), but 2.7x GPT-6.1 Sol (max). After the discount ends, this will rise to $3.98 (~1.2x GPT-6 Astra (max)).
Gemini 4 Argon is currently being rolled out to selected users and is not publicly available. The 50% discount is an initial promotion. Google has not yet confirmed the promotion end date
Key benchmarking results for Gemini 4 Argon with high reasoning:
➤ Google returns as one of the top three labs on intelligence: Gemini 4 Argon (high) scores 53 on the Artificial Analysis Intelligence Index, matching GPT-6 Astra (max, 53) and 1 point ahead of GPT-6.1 Sol (max, 52). This is 23 points above Google’s previous non-Flash model, Gemini 3.1 Pro Preview (30) and 12 points ahead of Gemini 3.8 Flash (high)
➤ Launch discounts of 50% make Gemini 4 Argon competitive on Cost per Task: At current discounted pricing, Gemini 4 Argon (high) costs $1.99 per Intelligence Index task, 60% of GPT-6 Astra (max, $3.26) for a comparable level of intelligence. This cost efficiency is driven by lower token prices, rather than reduced token use, with Gemini 4 Argon averaging 62k output tokens per task, compared with 27k for GPT-6 Astra (max). Google has not yet confirmed the promotion end date, but on standard pricing, Cost per Task will increase to $3.98
➤ Stronger agentic performance: Historically a weaker area for Gemini models, Gemini 4 Argon shows improvements across agentic evaluations. It ranks #1 on AutomationBench-AA at 77.5%, 6 points ahead of Claude Sonnet 5.5 (max, 71.3%). On Terminal Bench 4, Gemini 4 Argon achieves 57%, a +53 point improvement from Gemini 3.1 Pro Preview, only behind Claude Sonnet 5.5 (max, 64%), Claude Opus 5.5 (max, 60%) and GPT-6 Astra (59%). On AA-Briefcase, it reaches 1494 Elo. This is driven by a 65% rubric pass rate, the highest we have recorded, but lower Analytical Quality (1576 Elo) and Presentation Quality (1308 Elo)
➤ Lowest hallucination rate among leading models: On AA-Omniscience, Gemini 4 Argon has a 15% hallucination rate, the lowest of any model scoring 45+ on the Intelligence Index, compared with 51% for GPT-6 Astra (max) and 54% for GPT-6.1 Sol (max). This means Argon is much more likely to acknowledge when it does not know an answer rather than guess incorrectly. On accuracy, Gemini 4 Argon scores 50%, a 5 point decrease from Gemini 3.1 Pro Preview, and 13 points below GPT-6 Astra (max, 63%). With this slightly lower accuracy, its overall AA-Omniscience score of 42 remains in line with GPT-6 Astra (43) and GPT-6.1 Sol (42)
Key model details:
➤ Context Window: 1M tokens
➤ Multimodality: Text, image, video, and speech input, with text output
➤ Pricing: $4/$20 per 1M input/output tokens at standard pricing, currently discounted 50% to $2/$10. Cached input tokens receive a 95% discount ($0.10 per 1M at discounted pricing), up from 90% on Gemini 3.8 Flash
➤ Long Decode Continuation: We tested Gemini 4 Argon with Long Decode Continuation, a new Gemini API feature that pauses long responses and resumes them across follow-up calls. This lets reasoning run up to 1M output tokens without request timeouts
OpenAI got computer use to actually work with GPT-6 Astra just in time for taxes to be due. Well done 👏
George Cameron retweeted
Safety refusal reporting is now available in the Artificial Analysis Coding Agent Index
In our latest Coding Agent Index v1.5, we’ve introduced safety refusal reporting to help explain model behavior and score differences. A safety refusal occurs when a provider or model declines to start or continue a task on safety grounds.
An agent may fall back to another model to continue, or stop the attempt with a block.
Claude Fable 5.1 had the highest fallback rates in both Claude Code and Devin Fusion, with fallback attempts accounting for 8.8% and 7.1% of the Index's weight, respectively; these results therefore include the fallback models' performance.
Refusal variability, harness context buildup, effort settings, and retry strategies can all affect the observed rates.
George Cameron retweeted
Independent evaluation of AI models, across both capability and safety, is essential. We welcome Dario’s call to give independent evaluators greater access to frontier AI models.
For nearly three years, Artificial Analysis has been building benchmarks and infrastructure to independently measure AI capabilities. We have supported pre-launch benchmarking of frontier models with almost every major AI lab. We are continuing to see jumps in capability across every dimension we measure.
The world needs a vibrant ecosystem of independent AI evaluators. We are going to keep working to build it!
We Must Pace the Frontier: I’ve written a new essay on why the AI industry should slow down, with a three-part plan for doing so.
Anthropic is unilaterally committing to the first of these steps. We’ll provide third-party evaluators with permanent, employee-level access to our systems, so that they can verify adherence to our safety measures, report on incidents, and assess models’ alignment during training.
You can read the full post here: darioamodei.com/post/we-must…
George Cameron retweeted
Last week the Intelligence Index vs Cost Pareto frontier moved out substantially. Claude Fable 5.1, Muse Spark 1.3, and GPT-6 Astra each set a new point in efficient intelligence
Link to analysis: artificialanalysis.ai/#intel…
George Cameron retweeted
Announcing Artificial Analysis Intelligence Index v4.3, upgrading Terminal-Bench to 4.0 and adding AutomationBench-AA, an agentic workflow automation benchmark with a private test set. This is a continuation of our rollout of Intelligence Index v5
Changelog (Index v4.2 → Index v4.3):
➤ Terminal-Bench: 2.1 → 4.0, completing our upgrade to the latest version of Terminal-Bench
➤ Replacing 𝜏³-Banking with AutomationBench-AA, our implementation of Zapier's business workflow automation benchmark
We are continuing to prioritize keeping Intelligence Index as useful as possible by bringing forward a subset of the changes we had planned for Index v5. Each change in v4.2 and v4.3 stands on its own merits and brings the Index closer to real-world problem solving, adds more private test sets to prevent gaming, and reduces saturation
Intelligence Index v4.3 raises the difficulty of agentic coding tasks and broadens the types of agentic workflows tested. Because we use a held-out test set for AutomationBench-AA, in collaboration with @zapier, the weight assigned to evaluations with private tasks or answers increases from 40% to 45%. Category weights are unchanged from v4.2: Agents 30%, Coding 20%, General 30%, Scientific Reasoning 20%
Detailed changes:
➤ Upgraded Terminal-Bench 2.1 to 4.0: 66 multi-step tasks testing agents on tasks run in agent sandboxes driven via the terminal, including tasks involving software engineering, machine learning, science, and operations. The 4.0 update recalibrates compute and time allowances, and improves task instructions and verification. We have changed from the Terminus 2 harness to mini-SWE-agent, a minimal, model-agnostic harness. We will also be updating our Coding Agent Index, where we test model and harness pairs, to include Terminal-Bench 4.0 soon
➤ Replaced 𝜏³-Banking with AutomationBench-AA: Our implementation of Zapier’s AutomationBench tests agents on 657 business workflows across simulated applications such as Gmail, Slack, Salesforce, and Jira. Agents must complete task objectives while following business rules. AutomationBench-AA uses Zapier’s private set of 657 tasks, and is built on v1.0.6
Key results:
➤ Claude Fable 5.1 and GPT-6 Astra lead the Intelligence Index: Both Claude Fable 5.1 (max with fallback) and GPT-6 Astra (max) score 53 on Intelligence Index v4.3, followed by Claude Opus 5 (max, 51), Claude Fable 5 (with fallback, 50), Muse Spark 1.3 (max, 48) and GPT-5.6 Sol (max, 47)
➤ GLM-5.3 and Kimi K3 continue to lead open weights models (both at 44): GLM-5.3-Flash (42) is the third strongest open weights model, followed by Qwen3.8 2.4T A95B (40) and DeepSeek V4 Pro 0813 (max, 36)
➤ 4 labs occupy the Intelligence vs. Cost per Task Pareto frontier: OpenAI occupies the majority of the cost-efficiency frontier, with all five reasoning efforts of the recently released GPT-6 Astra offering the lowest Cost per Task at their respective levels of intelligence. Claude Fable 5.1 (xhigh, max, 53), GLM-5.3-Flash (42) and MiMo-V2.5-Pro (26) round out the rest of the frontier
We are laser focused on being useful to developers making decisions!
Intelligence Index v4.2 is an acceleration of our v5 roadmap, which is focused on long-horizon tasks that better resemble real-world problem solving, a greater proportion of private test sets to prevent gaming, and reducing saturation. Expect further updates as we continue our v5 rollout!
We are accelerating our rollout because the Intelligence Index needs to measure what AI is capable of to be useful to developers making decisions.
The key changes are adding AA-Briefcase, our long-horizon agentic knowledge work benchmark with a private test set, and GDP.pdf, which measures long-context reasoning across thousands of realistic documents. Both improve realism and AA-Briefcase increases the proportion of private test sets. We've also removed GPQA Diamond: it provides little signal for frontier models given its saturation, and its multiple-choice format doesn't reflect real tasks in those domains.
Link to our methodology: artificialanalysis.ai/method…
Proud of this update! More coming soon
Announcing Artificial Analysis Intelligence Index v4.2. We are accelerating elements of our upcoming v5 release with interim updates to keep pace with the frontier. Index v4.2 has more complex and realistic tasks, and more private test sets to prevent gaming
Intelligence Index v4.2 changelog:
+ AA-Briefcase, our agentic knowledge work evaluation with a private test set
+ @HelloSurgeAI's GDP.pdf, long context document reasoning across 4,592 PDF pages
- GPQA Diamond, an exceptional scientific reasoning evaluation that has now been saturated
… plus greater weighting on held-out test sets to prevent gaming, and grading infrastructure upgrades to increase robustness
This update brings the Index closer to real-world use cases with more challenging, complex and realistic tasks and private test sets to prevent gaming. We have been planning and building elements of Index v5 for months - it’s been 8 months since we launched Index v4 in January.
We have deliberately held back updates to keep the Index stable through recent major model launches. However, with the frontier moving so quickly in the past weeks, we feel it is important to deliver an immediate interim update to ensure our Index remains as relevant and useful as ever to users.
Beyond this interim update, our team is hard at work on v5 of the Index. We are planning more incremental releases in the near future. Stay tuned!
Intelligence Index v4.2 changes in detail:
➤ Adding AA-Briefcase: Our in-house evaluation with a private held-out test set, AA-Briefcase tests models on realistic agentic knowledge work tasks in complex projects built by industry experts. Models are evaluated on multi-week knowledge work projects, each with many linked tasks and thousands of input source files. AA-Briefcase combines rubric and pairwise grading to evaluate verifiable task success, analytical quality, and presentation quality, giving a holistic view of overall agentic capability in knowledge work.
➤ Adding GDP.pdf: Created by @HelloSurgeAI, GDP.pdf evaluates single-turn professional document reasoning across 100 PDFs and ten domains. Models must synthesize evidence distributed across 4,592 pages, including text, tables, charts, footnotes, and exclusions. Responses are graded against 1,275 expert-authored atomic criteria; the headline All-pass Rate credits a task only when every criterion is satisfied.
➤ Weighting to measure real-world use and prevent gaming: 40% of our Index weighting is now private, held-out test sets - double the figure from v4.1. Held-out data includes AA-Briefcase, AA-Omniscience, and solutions for CritPt. This reduces the ability for labs to game evaluations. The held-out percentage will increase further in Index v5.
➤ Improving our grading infrastructure: In AA-LCR v1.1, we have added a grading system prompt and corrected errors and ambiguities in answer keys, improving scoring accuracy. For GDPval-AA v2 and AA-Briefcase, we have improved our sampling and re-anchored the Elo scale, making ratings more stable as new models are added. For SciCode we have improved robustness of grading sandboxes to ensure slow but correct code does not count as a failure.
Key results:
➤ Anthropic and OpenAI lead the Index: Anthropic’s Claude Fable 5.1 leads the Index, followed by OpenAI’s GPT-6 Astra, which shows a 4pt gain over GPT-5.6 Sol. Meta is the third-ranked lab on the leaderboard, followed by SpaceXAI, Moonshot/Kimi, Z AI, and Google
➤ Cost per Task Pareto frontier shared by four labs: Anthropic, OpenAI, Meta and Z AI occupy the updated Cost per Task Pareto frontier
➤ GPT-6 Astra dominates the output token Pareto frontier: GPT-6 Astra is more token efficient than almost every other model near the intelligence frontier, with Claude Fable 5.1, Grok 4.5 and Gemini 3.5 Flash-Lite at either end of the curve (excludes models below 25 on the Index)
90% of AI forward organizations know they need custom benchmarks, but fewer than 5% have actually built one
This is because benchmarks that meaningfully test frontier models are incredibly hard to build. It takes dedicated research focus for us at Artificial Analysis.
Optima makes custom benchmarks accessible to anyone by leveraging Artificial Analysis' research, experience and platform. Really excited to release it to today
We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform
Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task.
We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow:
➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or @huggingface, or import agent traces from platforms including @arizeai, @braintrust and @langfuse. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you
➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released
➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set
➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case
Ahead of launch, here are examples questions our beta testers answered with Optima:
➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent?
➤ Which model best matches the writing style of lawyers for my legal agent?
➤ Which model can best identify different elements in my custom image dataset?
Optima is available today. Build your own benchmark at artificialanalysis.ai/optima
George Cameron retweeted
We’re launching Optima. Now anyone can create a custom benchmark for their use case, leveraging Artificial Analysis’ leading research and platform
Building and running benchmarks is difficult. We have distilled Artificial Analysis’ research and experience developing benchmarks into Optima, a new platform for benchmarking models on your own workloads and comparing performance, speed and cost efficiency. Optima allows you to find the best model for your task, or an equally performant alternative to your current setup at 10x lower cost or time per task.
We’ve integrated Artificial Analysis' research and experience in benchmarks across the Optima workflow:
➤ Build benchmarks based on your own data and use cases: There are three ways to build a benchmark with Optima. Upload an existing evaluation dataset from your own files or @huggingface, or import agent traces from platforms including @arizeai, @braintrust and @langfuse. Install the Optima skill to build a benchmark using context from your coding environment and previous sessions. Or simply describe your use case and provide example inputs and outputs, and Optima will build the benchmark for you
➤ Run across the latest models: Run the same benchmark across leading models in a single click, and keep your leaderboard up to date as soon as new models are released
➤ Bring Artificial Analysis grading to your own benchmark: Evaluate responses against objective rubric criteria or using the same pairwise judging approach used for Artificial Analysis benchmarks including GDPval-AA and AA-Briefcase. For pairwise judging, select your preferred responses from a sample and Optima uses those preferences to rank models across your test set
➤ Compare performance, cost and time efficiency: Optima measures more than model performance. Cost per Task and Time per Task are tracked alongside benchmark scores, with category-level results and support for custom metrics, allowing you to compare the tradeoffs between models for your specific use case
Ahead of launch, here are examples questions our beta testers answered with Optima:
➤ Which model can save me 10x the cost without a meaningful decrease in quality for my finance & accounting agent?
➤ Which model best matches the writing style of lawyers for my legal agent?
➤ Which model can best identify different elements in my custom image dataset?
Optima is available today. Build your own benchmark at artificialanalysis.ai/optima
George Cameron retweeted
Announcing AA-AnalystAgent, our new agentic benchmark for quantitative analysis on real-world spreadsheets & documents. Claude Opus 5 leads at 54%, followed by GPT-5.5 at 50% and Claude Fable 5 at 49%
In real analyst roles, professional judgment and expertise are as important raw quantitative capabilities. AA-AnalystAgent tests this and requires models to interpret sources, decide which exceptions and caveats apply, and settle on a methodology to successfully complete a task.
Because handing analyst tasks to agents requires not just correct answers but consistent ones, AA-AnalystAgent runs each task five times and reports pass^5 as its headline metric (we call this ‘pass-all-5’). pass^5 means models must get a task correct every time it tries across five independent attempts to pass.
AA-AnalystAgent overview:
🔢 80 questions across 14 business and scientific domains, including healthcare expenditure reports, trade and commodity statistics, hydrology and weather data, government appropriations, energy cost models, financial models, environmental reporting, and project schedules
🧠 Five workflow buckets from across real analyst work: source lookup and diagnosis, filter and total, ratios/trends/sensitivities, P&L modeling, and cash/balance sheet/valuation modeling
Methodology details:
🤖 Agentic harness: each task is solved by an agent running in our open-source Stirrup reference harness, with tools for code execution, web fetch, image viewing, and answer submission
📊 Scoring: each task is run five times per model, with final answers compared to reference solutions by an equality checker. Three metrics: pass^5, pass@1 (average pass rate), pass@5
🔒 Privately-held question set to limit contamination risk. Two example questions from California Medicaid expenditure reports are publicly shown on the methodology page with full prompt and source material
Key findings:
🥇 @AnthropicAI's Claude Opus 5 (max) leads at 54%, followed by @OpenAI's GPT-5.5 (xhigh) at 50% and Claude Fable 5 (max, Opus 4.8 Fallback) at 49%. Anthropic holds three of the top five places, and the top three are separated by three net tasks out of 80.
🎯 Reliability separates the top of the leaderboard more than raw capability: GPT-5.5 (xhigh) has the highest pass@1, but Opus 5 leads on pass^5 because it repeats what it gets right.
🔍 Committing early to a wrong interpretation is the most widespread way models fail, appearing in 57% of the failures we classified.
💰 The price of a given score varies enormously: Claude Sonnet 4.6 and @Xiaomi's MiMo-V2.5-Pro both score 20%, at $1.34 and $0.05 per task.
🥈 @Kimi_Moonshot's Kimi K3 (max) is the top open weights model at 39%, 15 points behind the closed frontier.
AA-AnalystAgent launches as a standalone leaderboard and is not part of the Artificial Analysis Intelligence Index.
See below for further detail ⬇️
