the obvious new bottleneck is absorbing information and clearly communicating what to do next

new achievement unlocked? running an eval on a model, it aces a tricky task, then it starts to celebrate and summarize... This request was declined because it could enable cyber harm.

at zapier we got access to opus 5.5 a while back to run automationbench and found it to be exceptional at 40% on max, just a hair below gpt 6 astra at 41.4% and well above fable 5.1 at 31.4%. some reflections below: the writing is certainly better and less fried -- the efficiency gains are easy to feel. i'll admit its a little unnerving to see such complex bash and python scripts just flying by, but the elegance in which it reads a token from the clipboard, validates it, and persists it safely is very impressive. anecdotally, an earlier model would take many turns to do that, if it did it at all -- opus 5.5 does it one sleek pass. crazy! while i am speculating, it seems pretty clear that the labs are incentivizing efficiency (# of turns/tokens) in their training. you can certainly feel it when you come back to your tab to a killer answer and only 90k tokens spent when you'd have expected 200k+ or more! i think this places some pressure on coarse mcp tools which lack the compactness that access to bash and rg/head/tail/jq/etc. afford. seems like code/bash mode really is the way!

tl;dr -- jev is a very useful paradigm which has been hiding in plain sight. also, pour one out for log probs disappearing from so many new closed inference apis... 😢 prompting llms with max_tokens: 1 w/ log probs for fast, parallelized judgments may be an old pre-structured outputs trick gaining new life, but i'll admit i'm rather surprised at how flexible and useful this pattern appears. there's a lot more depth here than i expected -- the demos are quite impressive! back in the golden days of early llms circa 2023, i recall experimenting with fine tuning oai's ada to output either a 1 or a 0 token, and inspecting the log probs to roughly gauge confidence. at zapier we were exploring this around field mapping -- matching output fields from a prior step to input fields in another. seems like training a base model specifically to return calibrated probabilities takes that quite a bit further. it's so cheap that you can ask tons of speculative questions and just throw away the answers. add a hierarchy of follow-up questions and you can really tune the plinko machine. quantity truly has a quality of its own. with short questions and shared context, the rough math works out to ~60 field judgments per penny with fine-tuned ada versus potentially ~11k with jev batching -- amazing! what other interesting paradigms are lurking in plain sight?

gpt 6 luna (if it happens) will presumably be an absolute beast at max effort.

okay, so astra just set a new high on automationbench: 41.4% of business workflows completed correctly. so how is it better? and what is automationbench? well, we measure that by putting the model in simulated apps, giving it a job, and checking the resulting records and messages with code. the hard part is building a test that catches "updated the wrong customer" or "ignored the hold" while still letting the agent figure out its own way to do the work. more or less, you build an small environment for an agent. start with a known set of crm records, emails, spreadsheet rows, calendar events, etc. give the agent tools that behave like those apps' apis. underneath, it's structured state: sending an email adds a message to the sent mailbox, updating a contact changes a field on that record. nothing goes to an actual customer. each run starts with its own copy of the environment, so one model's changes don't affect another's. then you give it a task. say the spreadsheet has verified job titles and you want the crm brought up to date. update the contacts, mark the spreadsheet rows reconciled, and send the summary to the audit inbox specified in the company's procedure. the agent has to find the procedure and the data itself. it gets a keyword search over api documentation and a tool for making requests with a method, url, and body. there's a budget of 50 turns, and it can make several tool calls in a turn. you can make this quite hard without changing that basic request. put two people with the same name in different accounts. leave an outdated title in one source. mark a row verified, then put a later message in the inbox saying that title is disputed and shouldn't be changed yet. put another record under a security freeze. now the agent has to work out which records the request actually applies to. "update everyone in the spreadsheet" won't do it. the critical information need to be discoverable, though. if the correct answer depends on a fact we never put in the environment, that's our bug. same if the api can't retrieve the relevant message, or the instructions contradict each other without a way to tell which one governs. making a task difficult is easy. making it difficult for a reason that tells you something about the model takes significantly more work. scoring is ordinary, deterministic code that inspects the final state. for the contact task, the checks are along these lines: the eligible contacts have the expected titles, their spreadsheet rows are marked reconciled, and the audit inbox received the required summary. but you also check that the held contacts kept their original titles and statuses. otherwise a model could update everyone and pass just because the two intended updates happened somewhere in there. that's an actual difference we saw between astra and sol. astra updated two eligible contacts; sol updated four, including two it should have left alone. sol had even retrieved the security-freeze message for one of them before making the change. a grader that only checked whether the requested updates happened would miss that failure. crucially -- you have to check the unwanted changes too. the same applies to messages. checking that the right person received an email isn't enough if the agent also sent it to twenty wrong people. these negative checks are part of the task, not an optional safety score off to the side. every scored check has to pass for the workflow to count as complete. we keep partial credit to help inspect failures, but 41.4% means complete tasks, not the average fraction of steps the model got right. we don't prescribe the whole sequence of calls. the agent might find the data a different way, batch some updates, or correct an earlier mistake. if it leaves the required state behind, that should pass. this also means the benchmark only sees what its checks cover. a deterministic grader can be consistently wrong! we audit the tasks and use separate hint-assisted runs to help check that they're solvable; the hints aren't part of the scored runs. the simulator and the grader both need scrutiny when a model fails. the public task set is available for experimentation. the headline scores use a separate, harder held-out set, so don't mix those numbers. a few model developers (meta, z.ai, etc) are reporting public set scores which aren't verified by us. and none of these scores tell you that the model will succeed on that percentage of work in your company. they tell you how often it completed this particular set of workflows under these conditions. we're actively working on automationbench 2, something that will be significantly trickier...

fifth, it's way more persistent. it doesn't quit, a lot of times other models just kind of randomly stop. astra doesn't. sixth, its pretty good at coordinating work -- running a few together is fun to watch. 3/?

third, there are way more moments of "wow, it really gets it", it does more effective research before starting to do work, and has way better judgement. fourth, it does an impressive amount of "creative grinding". clever micro benchmarks, skipping helpers for direct data. 2/?

In the age of agents, it's surprising how important information accessibility becomes. If conversations and thinking is hidden away, it's not very useful, is it? Try this in before replying to your next DM: "can we post this somewhere public?"