📢 New Paper Can LLM Agents Infer World Models? We turn a well studied automata learning task into an agent benchmark, where a hidden automaton serves as a world model that an agent must reconstruct using query tools. Our findings show that SOTA LLM agents can sometimes perform non trivial interactive discovery, but remain far less robust and efficient than classic automata learning algorithms. Beyond overall performance, our analysis reveals recurring failure modes in planning, reasoning, and using accumulated information over long contexts. 🔹 Think this task is easy? Try it yourself: reefmenaged.github.io/Agenti… 🔹 Watch an LLM tackle an automaton of your choice: agentic-automata-learning.on… 🔹 Read the paper: arxiv.org/pdf/2606.16576 Grateful to my amazing co-authors for making this work possible: @GiliLior @ravfogel @roeeaharoni @GabiStanovsky

New version of our paper is out! In v2, we evaluate Agentic Automata Learning with more advanced harnesses and scaffolds, including an OpenHands coding harness and a ReAct style state tracking scaffold. We find that agents can identify an algorithm for the problem and even implement it with coding tools. Yet in the interactive task, without coding tools, they make recurring errors in query planning, evidence integration, and hypothesis construction, reaching only 30% success on the hardest DFAs. ReAct style state tracking improves performance, but agents still suffer from these recurring failures, leaving substantial room for improvement compared with classic algorithms in both success rate and interaction efficiency. Check out the new version and read more 👇 arxiv.org/abs/2606.16576v2

I think a meaningful part of Astra’s recent jump on ARC-AGI-3 comes from a technical change: it was also evaluated with a Provider Adapter harness, a richer agentic runtime that preserves state across steps, including hidden reasoning state and context compaction. The key point is that users can’t fully recreate this themselves. You can build an agent loop, memory, and context management around an API, but the model itself remains frozen. The model provider, on the other hand, can do RL and post-training with the model running inside the same runtime it will use at inference time, teaching it how to actually use that persistent state. That’s why I think this is the right direction for the industry: frontier models should be shipped together with their agent and state-management mechanisms as one system, just like with GPT-6 Astra. I don’t want just a stateless input-output API that I have to build an agent around. I want a native multi-turn system where the provider maintains the working state and the model has been trained to use it.

An advanced harness doesn’t just add capabilities to an LLM agent, it multiplies them: there are many problems the model learned how to solve during training, but solving them requires long-horizon strategies and maintaining a coherent world model across many steps. With a basic harness, the model is limited in its ability to turn that problem-solving knowledge into consistent execution.

Perfect, I can retire now

Great blog post covering many questions that come up when building an agent benchmark

A phenomenon I keep noticing across recent papers is that LLM agents terminate interactions prematurely without fulfilling the task, even when explicitly told they can continue interacting to gather more information. We see this in benchmarks like MT-PINGEVAL [1], where dialogue agents rush to guess instead of using extra turns to uncover critical context, and ProgramBench [2], where coding agents propose the hidden program before exhausting the tool that provides observations on the program. This persistent agentic limitation warrants deeper research! [1] arxiv.org/pdf/2602.24188 [2] arxiv.org/pdf/2605.03546

An interesting project we’ve been working on: From Words to Wins: Analyzing Pre-Tournament Interviews to Predict Tennis Tournament Success 🎾 We built a model that combines interview representations learned by an encoder with structured features such as player ranking, and used it to predict whether a player would perform at least as well as their recent average. In our setting, we found that adding the interviews did not improve prediction performance compared with a model based only on structured features. I think the most interesting question still remains open: with the right architecture and data at scale, should we expect to see an improvement? Or is there simply not enough predictive information encoded in these interviews to help forecast tournament performance?

$100 for a single agent rollout is insane. Agent evals are becoming so expensive that cost itself is starting to limit who can do meaningful research. Model providers should create programs where researchers can apply to run novel evaluation benchmarks at heavily discounted rates. It’s in everyone’s interest: researchers get affordable access to frontier models, while labs get independent researchers testing their models and uncovering limitations they might otherwise miss.

I agree that evaluating agents with the minimal toolset required to solve the task is the right way to measure their actual capabilities, rather than the strength of the harness around them. For example, in code tasks, adding internet access may turn the benchmark into a test of retrieval if the solution already exists online. With only the tools necessary to solve the task, we get a cleaner signal of the model’s coding ability and how well it generalizes to new problems.

Reviewers sometimes seem to assume unlimited compute, time, and annotation budgets. In reality, you cannot check everything and have to prioritize. If it were their own paper, they would probably make the same tradeoffs.