PhD Student @ELLISforEurope @MPI_IS, previously @SakanaAILabs Prior works: FutureSim, Measuring Long Horizon Execution
Joined August 2019
- Tweets177
- Following924
- Followers795
- Likes2.4K
Pinned Tweet
Introducing FutureSim: where we replay a temporal slice of the web and let agents forecast real-world events over time 🔮🌎
FutureSim replays the web day by day. Agents start on Jan 1, 2026 (past their knowledge cutoffs) with date-gated access to real news articles and forecast on real-world events resolving over the next 90 days. Around 244K new articles stream in during the simulation. Agents decide which questions to answer, what to search for, and when to advance to the next day 🤔
We evaluate frontier models in their native harness. GPT 5.5 (Codex) leads at 25% acc, followed by Opus 4.6 (Claude Code) at 20% 📈 Open weight frontier models have a significant gap to catch up, with DeepSeek V4 pro at 13%, GLM 5.1 at 10%, and Qwen3.6 Plus at 5%
On some questions that have a parallel @Polymarket market, we find that GPT 5.5 in our simulation sometimes beats the crowd aggregate, like in the Super Bowl LX ($704M traded) market 💰💸
FutureSim serves as a test bed for evaluating a lot of important agentic capabilities
> Adaptation: how agents adapt beliefs over time, and handle new incoming information and environment feedback
> Memory: how agents make the best use of external memory to store persistent insights and handle context limitations over a thousand tool calls
> Search: how agents find relevant information over thousands of articles streaming in
> Inference scaling: how agents benefit from scaling inference compute
More cool insights and deep dives in our paper 👇
Arvindh Arun retweeted
TB-Science 🤝 FutureSim! At the Frontier Data Summit by @SnorkelAI
Turns out we are all from the same region in Germany! In the order:
Tübingen <-> Metzingen <-> Stuttgart
Arvindh Arun retweeted
This paper shows why I am skeptical of the idea that frontier models can be easily replaced by cheaper ones (and why you should use the best models as orchestrators): Small improvements in accuracy & judgement compound over long-horizon tasks, creating huge gains in outcomes.
Arvindh Arun retweeted
Not many people know this, but FutureSim is super long-horizon: it has an order of magnitude more tool calls than DeepSWE or TerminalBench 4.0, for example!
openforecaster.github.io/fut…
I looked into the most popular long horizon benchmarks and the official number of reported tool calls per model run.
FutureSim admitted the highest number of tool calls among all the runs, far ahead of many other famous benchmarks!
You can find those long traces here: openforecaster.github.io/fut…
I will be in SF next week to present FutureSim at:
- Frontier data summit by @SnorkelAI on 8th October
- Two COLM workshops on 9th October
Excited to meet people working on long-horizon evals and RL envs. Please DM if you're around!
Arvindh Arun retweeted
FutureSim is a live backtesting simulation testing how models adapt at test time to new information and forecast outcomes. Runs span 90 simulated days, 5,000+ tool calls, and 12+ hours per model. From @ShashwatGoel7, @nikhilchandak29, and @arvindh__a. One of 25+ posters at Frontier Data Summit frontierdatasummit.ai/
Arvindh Arun retweeted
We'll be presenting FutureSim at:
- the Frontier Data Summit on Thursday October 8
- COLM workshops on Friday October 9
- NeurIPS in December
FutureSim is a live backtesting simulation testing how models adapt at test time to new information and forecast outcomes. Runs span 90 simulated days, 5,000+ tool calls, and 12+ hours per model. From @ShashwatGoel7, @nikhilchandak29, and @arvindh__a. One of 25+ posters at Frontier Data Summit frontierdatasummit.ai/
FutureSim will appear at #NeurIPS2026! 🇦🇺 now onto hoping we get a spot at sydney
Introducing FutureSim: where we replay a temporal slice of the web and let agents forecast real-world events over time 🔮🌎
FutureSim replays the web day by day. Agents start on Jan 1, 2026 (past their knowledge cutoffs) with date-gated access to real news articles and forecast on real-world events resolving over the next 90 days. Around 244K new articles stream in during the simulation. Agents decide which questions to answer, what to search for, and when to advance to the next day 🤔
We evaluate frontier models in their native harness. GPT 5.5 (Codex) leads at 25% acc, followed by Opus 4.6 (Claude Code) at 20% 📈 Open weight frontier models have a significant gap to catch up, with DeepSeek V4 pro at 13%, GLM 5.1 at 10%, and Qwen3.6 Plus at 5%
On some questions that have a parallel @Polymarket market, we find that GPT 5.5 in our simulation sometimes beats the crowd aggregate, like in the Super Bowl LX ($704M traded) market 💰💸
FutureSim serves as a test bed for evaluating a lot of important agentic capabilities
> Adaptation: how agents adapt beliefs over time, and handle new incoming information and environment feedback
> Memory: how agents make the best use of external memory to store persistent insights and handle context limitations over a thousand tool calls
> Search: how agents find relevant information over thousands of articles streaming in
> Inference scaling: how agents benefit from scaling inference compute
More cool insights and deep dives in our paper 👇
Arvindh Arun retweeted
The mysterious Ox Alpha is now on FutureSim!
Better than most chinese frontier models, but fails to show strong test-time adaptation.
If released openly, it might be one of the strongest and cheapest open-weights model.
Meanwhile GPT remains the 👑
Recent models write so bespoke, it’s jarring. It’s probably because text is doing three jobs at once: communication, computation, and interaction with the world.
When models are eventually trained with physical ways to interact with the world, we might get 4o writing back!
Arvindh Arun retweeted
We can finally talk about it:
We found a way to extract hidden reasoning of frontier models using a vulnerability in the APIs of every frontier AI company.
We verified that our reasoning token count matches billed API thinking tokens 1:1 for most of the prompts we queried.
there’s still a lot of alpha in harnesses/scaffolding, great release from @PrimeIntellect!
FutureSim v1.1 is out! Latest models tested on better and newer data 🔮
🚨 FutureSim Update 🚨
With new models coming out, we updated our test set to replay world events from April to June'26.
And GPT-5.6-Sol is the new leader performing over 15K tool calls in a single run and taking > 24hrs to complete!
Both Fable-5 and GPT-5.6 start with similar accuracy but GPT-5.6 shows strong test-time adaptation while Fable and Opus barely improve.
Even GPT-5.5 with weaker priors surpasses Fable!
Arvindh Arun retweeted
🏆 Congrats to @ShashwatGoel7, @nikhilchandak29, and @arvindh__a, Best Paper winners at our forecasting workshop at #ICML2026, and to runner-up @sirbayes 🥈.
Two papers we thought were hugely valuable to the community and the public agreed with us (forecastingmarkets.org/marke…)! 📈
FutureSim wins the Best Paper Award at the AI Forecasting workshop at ICML 🔮 I’ve officially hit the fomo trifecta this time, not being in Seoul 😮💨
Introducing FutureSim: where we replay a temporal slice of the web and let agents forecast real-world events over time 🔮🌎
FutureSim replays the web day by day. Agents start on Jan 1, 2026 (past their knowledge cutoffs) with date-gated access to real news articles and forecast on real-world events resolving over the next 90 days. Around 244K new articles stream in during the simulation. Agents decide which questions to answer, what to search for, and when to advance to the next day 🤔
We evaluate frontier models in their native harness. GPT 5.5 (Codex) leads at 25% acc, followed by Opus 4.6 (Claude Code) at 20% 📈 Open weight frontier models have a significant gap to catch up, with DeepSeek V4 pro at 13%, GLM 5.1 at 10%, and Qwen3.6 Plus at 5%
On some questions that have a parallel @Polymarket market, we find that GPT 5.5 in our simulation sometimes beats the crowd aggregate, like in the Super Bowl LX ($704M traded) market 💰💸
FutureSim serves as a test bed for evaluating a lot of important agentic capabilities
> Adaptation: how agents adapt beliefs over time, and handle new incoming information and environment feedback
> Memory: how agents make the best use of external memory to store persistent insights and handle context limitations over a thousand tool calls
> Search: how agents find relevant information over thousands of articles streaming in
> Inference scaling: how agents benefit from scaling inference compute
More cool insights and deep dives in our paper 👇
catch maya presenting our work at KnowFM workshop at #ACL26!
Now you can have importance scores for nodes and subgraphs in your LLM GraphRAG pipeline!
“Ex-GraphRAG” at the ACL 2026 Towards Knowledgeable Foundation Models Workshop. 😎
In this work, we study interpretable evidence routing for graph-augmented LLMs. We show that standard GraphRAG encoders obscure which retrieved entities actually influence the LLM, and replace them with M-GNAN, an intrinsically decomposable encoder that provides exact node-level importance during the forward pass.
M-GNAN is now available in PyTorch Geometric and can be used directly in the G-Retriever framework by simply choosing it as the G-Retriever GNN encoder:
pytorch-geometric.readthedoc…
Congrats to all authors! @arvindh__a Yoav kor-Sade, Rishi Puri
FutureSim is even more accessible (both for analysis and new runs) now! It’s now hosted @OpenReward, you can start playing around with it in just a couple of steps 🔮
Major FutureSim updates!
🥽 Visualized trajectories: extremely long horizon, 1000s of actions per run and dozens of compactions, with summaries! Goldmine of interesting agent behaviors across Fable 5, GPT 5.5, GLM 5.2 etc!
📈 @OpenReward integration! You can now use FutureSim, incl. on your own data, with our favourite standard! The @GenReasoning was super supportive in making this happen!
Some cool observations in 🧵
Arvindh Arun retweeted
New blogpost: What does your benchmark actually measure?
I argue what evals end up measuring is the simplest way to improve performance on them. When building a new benchmark, we are often too focused on minimizing scores at launch.
This often makes benchmarks too easy to increase scores on, without reflect improvements in the underlying capability it intended to measure.
Instead, I find it useful to think about, and apply, all the ways someone could improve scores on a benchmark. This often reveals unintended shortcuts that were enough to make large eval gains.
Thinking about how a benchmark behaves under optimization pressure helps interpret results on it, and predict behaviors that will surface in future runs, as if it's successful, the community/models will try everything to hill-climb it.
Full post below:
opus 4.8 🤝 me after learning how to actually actually post-train models over the past few weeks: distillation
This quoted post is unavailable.
Arvindh Arun retweeted
So you were asking whether gains from coding would generalize to other domains?
We found GLM-5.2 to be no better than GLM-5.1 on FutureSim. The gap between open and closed-weights here is massive!
Also, despite Fable-5 being contaminated in Jan, it still scores only 27%.
Introducing GLM-5.2: Frontier Intelligence, Open Weights
- Significant improvements in coding and agentic tasks
- Strong long-horizon capabilities with a 1M context window
- Two levels of reasoning effort: GLM-5.2 (max) pushes the limits, while GLM-5.2 (high) strikes a strong balance between performance and token efficiency
- MIT-licensed open weights
- Same API pricing as GLM-5.1
Tech Blog: z.ai/blog/glm-5.2
Weights: huggingface.co/zai-org/GLM-5…
API: docs.z.ai/guides/llm/glm-5.2
Coding Plan: z.ai/subscribe
Chat: chat.z.ai
Our recent work on using multivariate additive networks (arxiv.org/pdf/2605.21994) to explain GraphRAG outputs is now officially a part of pytorch-geometric 🕸️
github.com/pyg-team/pytorch_…
catch @mayabechlerspei presenting our preliminary results in KnowFM at #ACL26 next month!
