Nick Papacon

Introducing the Artificial Analysis Decision Index v2: a new proprietary evaluation, updated benchmarks, and a headline metric that rewards model calibration We’ve released the second version of the Artificial Analysis Decision Index to better reflect real-world utility and guard against benchmark gaming. 45% of the Index is now built on private datasets, up from 25% in v1. Drex Max v1.1 from @NaceAI narrowly leads among decision models at 73, within 1 point of GLiDE v1.2 (@fastinoAI) and Mammoth 27B (@baselabs) Updates in Decision Index v2: ➤ Decision Score: We adopt a new headline metric that rewards a model for both correctness and calibrated confidence. Each question scores from 0 to 100: right answers score above 50, wrong answers below 50, and confidence in its top answer moves the score within each half ➤ AA-Paperclip (30%): We added a new private benchmark of 1,500 documents including resumes, receipts, invoices, and AI and machine learning research papers. Models make decisions and extract facts from each document and are scored jointly as a single decision, correct only if all 15 answers are right ➤ AA-Symmetry (15%): This evaluates whether models change their answers for equivalent questions worded differently. We test 120 questions with variations such as reordered options, swapped labels, and negated yes/no questions, then measure how consistent the model’s probabilities are across those variations ➤ AA-Omniscience-Decision v1.1 and HLE-MultiChoice v1.1 (15% each, down from 25%): AA-Omniscience-Decision now varies the wording of its “I don’t know” option across ten alternatives. HLE-MultiChoice adds an option to decline using the same ten phrasings We’ve also added EvalSafe-ONET-AA (15%), our implementation of @typesafeai’s ONET benchmark. We have retained When2Call from @NVIDIAAI in the Index at a decreased weighting of 10% (down from 25%), and removed WildGuardTest given its high saturation Key results: ➤ Three decision models sit within 1 point at the top of the Index: Drex Max v1.1 from Nace AI and GLiDE v1.2 from Fastino Labs both score 73, with Mammoth 27B from Base Labs scoring 72 ➤ Cost per 1M Tasks varies by more than 60x across decision models: Celeris-1.1 Decision ($14), Decider v1.2 ($68) and Drex Max v1.1 ($102) sit on the Decision Index vs. Cost Pareto frontier, while Clef costs $859. Token price is not the only cost driver: most decision models, including Drex Max v1.1 and Decider v1.2, bill a document’s input once for all its questions, while GLiDE v1.2 and Decider Flash v1.2 bill it for every question. This means GLiDE v1.2 costs $780, nearly 8x higher than Drex Max v1.1 despite only 33% higher list prices ➤ Language models can score higher, alongside higher Cost and Time per Task: Claude Haiku 5.5 (Max) scores 77 and GPT-6 Luna (Max) scores 76 on Decision Index v2, putting them ~3 points ahead of Drex Max v1.1. GPT-6 Luna (Max) costs $1,530 per 1M Tasks vs. $102 for Drex Max v1.1 (15x the cost) and takes 11.34 seconds per task vs. 275 milliseconds (~40x slower)

We've reset weekly limits for all Grok Bot users. Enjoy!

Grok Bot fixes for improved token efficiency and smarter behavior are underway

Generative UI for conversational agents. Combining @aisdk with Chat SDK and 𝚓𝚜𝚘𝚗-𝚛𝚎𝚗𝚍𝚎𝚛, you can go straight from LLM to UI on platforms like Slack and Teams. I believe this will become the dominant Human-Agent Interaction (HAI) paradigm: vercel.com/changelog/json-re…

Quantum technology could unlock new possibilities in drug discovery, materials design, and robotics. As commercial applications advance, businesses have an opportunity to build capabilities and gain a competitive edge. Explore the three horizons: mck.co/4BBwNkW

Background subagents in Claude Code now use less of your usage limit. A running subagent updates its status line every 30 seconds, and each update used to re-read its whole conversation. Now Haiku 5.5 writes it from short notes instead, so each update costs about 99% less.

I just canceled a $600/year SaaS subscription for an Opus-coded, in-house alternative. My external spend is now exclusively down to infrastructure: servers, email & payments. I think niche B2B SaaS is in real trouble.

Το Opso έρχεται σύντομα. Ένα app που σου λέει κάθε εβδομάδα σε ποιο σούπερ μάρκετ βγαίνει φθηνότερη η πρωτεΐνη σου, με αληθινές τιμές. Ίδιο καλάθι πρωτεΐνης, μία εβδομάδα: Lidl €56,58, My Market €80,18. Θα το χρησιμοποιούσες; Ψήφισε παρακάτω 👇

Chamath says his company's AI token bill is doubling every 45 days while output gains barely move. "I sat down with my CTO today, I said how are we doing on token spend. And he said the most incredible thing, he said right now, our token costs are doubling every 45 days. I said well what is the downstream productivity? And he said maybe 5% max." "So my costs are doubling every 45 days, my upside is essentially flat. He said honestly, what we're finding out is that you need to use a lot more tokens to get to this next iteration of improvement because we've effectively already asymptoted." "We're going to take a step back and try to figure out what to do. I don't know how many other companies will actually go through this reckoning now, but the point is everybody in the next three or four years will for sure go through it." "I suspect that if you can get out now, you should get out now before all of that starts to seep into the water table. Because I think that's probably what allows you to get out at a huge price and raise a huge amount of money." Interested in AI? Follow @SiliconValleyPX and never fall behind. I track ChatGPT, Claude, and how every new agent is changing how we work and think, then hand you the pure signals.