🔍 Introducing BackSearch. LLMs are increasingly asked to predict the future, but a good backtest requires a snapshot of the internet at a point in time. BackSearch allows LLMs to search the web as it was on a particular date. It’s great for: 🔮 Forecasting and prediction markets. 📈 Quantitative finance. 🌍 RL environments that simulate the world. 🥶 Freezing websearch for benchmark reproducibility We’re releasing a narrow slice of our index to begin with, focused on the news domain for 2026. Based on feedback we’ll open up more of our index in subsequent releases. 👇 Try BackSearch out with the link below.

Because of backtest variance, we also record a process-based rubric measure called "sophistication", which we track over time. This uses a human expert rubric to judge the sophistication of the strategies employed. GLM-5.2 shows impressive sophistication compared to other open models, although it still does not surpass the closed SoTA on this metric for any time this year. You can see the full leaderboard and more analysis here: gr.inc/releases/introducing-…

We evaluated recent open models on KellyBench. Here is what we found: 🏆 GLM 5.2 is new open source SoTA, but still loses -30% on average over 5 runs. 📈 We estimate GLM 5.2 is 6+ months behind the frontier based on KellyBench and internal quant evaluations. (Note: we have not evaluated Fable) 🌗 Kimi K2.6 slightly improves on Kimi K2.5 but still struggles at -60% average RoI. 🐈 Recent Mistral models struggle, obtaining mean RoIs of -78% and -99% respectively. Leaderboard link and more graphs below.

We've updated the leaderboard with GPT-5.5 results: gr.inc/releases/introducing-… TLDR: 30% strategy sophistication (new SoTA), much more efficient than GPT-5.4, but still in the red and systematically underperforming human approaches.

We’re also thankful that KellyBench was featured on the front page of the FT last month! (The British AI neuron has a superposition with football…)

Link here: gr.inc/releases/introducing-… We observe a positive trend in the quality of model strategies, but still weak saturation overall and systematic underperformance compared to human approaches. (Note: we impose no time limits for models to develop strategies; they can use as much inference compute as they see fit)

New models on KellyBench, the benchmark for long-horizon sequential decision making. 🎆 Claude Opus 4.7 - new state-of-the-art, with the highest strategy sophistication recorded on the benchmark (28.5%). But still loses -3.7% on average over five seeds. 🐋 DeepSeek V4 Pro - shows strong feature development capabilities, but scores poorly on overfitting rubrics which harms it in deployment (-47%). Link and more graphs below.

🌍 Last month we hosted the Complex Worlds Hackathon in London. Participants built an impressive range of environments, spanning synthetic game pipelines, arable farm management, dynamic vehicle routing, hospital triage, robotics, cybersecurity, and more. Congrats to our winners, Julie Huang and Khalid A!

One potential critique of our setup is that betting markets are highly efficient, and models might be doing the best they can with the data they are given. But we constructed expert rubrics and found models were unsophisticated and heavily underutilising available data. The best performing model Opus 4.6 achieved a sophistication score of only 32.6%. So even if the market is highly efficient, there is much room for models to improve. In fact we find a positive relationship between sophistication and ROI for the seeds evaluated.

📉 Frontier models struggle on KellyBench. We find there is a large gap between the ability to develop machine learning strategies, and executing and adapting them over long-horizons. Only Opus 4.6 and GPT-5.4 manage to avoid ruin. Every other model goes bankrupt at least once in the course of the season. The best performing models achieve an ROI of -11% and -14%.

The dominant evaluation paradigm is task centric, short-horizon, and sparse reward. KellyBench is a different beast. Models on KellyBench take 500-1000 tool calls to complete an episode, use hundreds of millions of tokens, and receive dense rewards each week. It's also expensive: one of our GPT-5.4 seeds on xhigh cost circa $2,000 to evaluate.

🎲 Introducing KellyBench, a new long-horizon evaluation for frontier models. KellyBench evaluates models within a year long sports betting market, a challenging and highly non-stationary environment. Every frontier model we test loses money. They struggle to design ML strategies, manage risk, and adapt as the world changes. Link and thread below.