Can the surrogate spot the rare mutations that make a kernel much faster than its parent? We call these discovery moments, the steps that break stagnation in a search. We mine 1,347 parent-child pairs and use the surrogate to predict the large improvements. Its scores are predictive, staying above the random baseline at every threshold and reaching higher precision for larger improvements. Precision still falls off quickly, so the surrogate works best for prioritizing which steps to verify on the GPU. GPU confirmation of a candidate discovery is still needed.

Does any of this help a real kernel search? We run a search with and without the surrogate on six tasks, holding the GPU-measurement budget per step equal for both. Without the surrogate, the search measures every candidate it proposes. With the surrogate, it proposes four times as many candidates per step and measures only the quarter the surrogate ranks highest. On four (TriMul, FP8 quantization, GDN ChunkFwd-o, GDN Recompute W/U) of the six tasks the surrogate search matches or beats the baseline's best kernel, often reaching it with far fewer measurements. On the other two (Cross-Entropy and GDN ChunkFwd-h) the baseline wins by 5% and 7%, and both are cases where the search saturates within its first few steps.

Does calibration training help only confidence, or also the ranking a search relies on? It also improves the ranking the surrogate produces, which is what a budgeted search relies on. Measuring the speedup found at each GPU budget, training with a Brier reward beats the base across most budgets, while correctness-only and CRPS training do not. CRPS accounts for how far off a predicted bin is and Brier does not, so CRPS might be expected to rank better. In our experiments Brier ranks better.

Can we train a surrogate to know what it doesn’t know? For reliable use, a surrogate should be uncertain about the runtime of candidates it understands poorly and confident about those it doesn’t. We test this by grouping predictions by stated confidence and plotting forecast error within each group. An ideal plot would be monotonically decreasing from left to right, meaning as the confidence goes up, the forecast error always goes down. Any deviation from non-monotonicity (if the line goes up, then down) means that the surrogate’s confidence was misaligned. The off-the-shelf model almost has this property. Training on correctness alone removes this property, making the model confident on candidates it gets badly wrong, and removing a clear relationship between accuracy and confidence. Adding a Brier calibration reward to the same correctness reward restores the property and improves on the base, so higher confidence reliably means lower error.

How does RL change the model's predictions? The untrained GPT-OSS-20B tends to call slow kernels faster than they are and the very fastest kernels slower than they are. RL redistributes the model's probability mass across the speedup bins. The confusion matrices show the base model's errors and how each reward (correctness or calibration) moves them. A perfect predictor would be dark only on the diagonals (green squares), meaning it always has predicted speedup = true speedup. There is a tradeoff. Training improves calibration but increases raw forecast error, and the reward chosen sets the balance between them. Essentially, the surrogate becomes less confident (as it should) after RL training for calibration.

How well do off-the-shelf LLMs do at this, with no training? We rank candidates by predicted speedup and check how fast the best kernel in the top-ranked fraction is, relative to the fastest kernel in the whole pool. For GPT-OSS-20B, measuring just the top 1% of the ranking finds a kernel within 30% of the fastest in the pool, and the top 50% gets within 6%. Gemini-3 Flash ranks best of the models we tested. Calibration varies much more across models. A model can rank candidates well while reporting confidence that does not match its accuracy, and we use RL to improve this.

Predicting an exact runtime from source alone is intractable. We instead predict how fast a candidate is relative to a reference kernel, sorted into eight logarithmically spaced bins. Logarithmic rather than linear bins make the target easier to predict, since the model only has to judge whether a kernel is much slower, near baseline, or much faster, which is easier than separating a 2x speedup from a 3x one. Given the reference kernel, the candidate, and the target hardware, the model reasons about how the candidate will execute and returns a probability distribution over those bins. Its confidence is the probability it puts on the bin it predicts, and a search can defer the low-confidence cases to the GPU. We then use RL to improve this distribution, rewarding the model both for predicting the correct bin and for reporting calibrated probabilities across all of them.

Can an LLM act as a selective model of a GPU during evolutionary search, by reasoning + forecasting a kernel’s runtime but deferring to a GPU when unsure? We produced 12k kernels + runtimes from evolutionary search, costing 400M reasoning tokens + 600 GPU-hours to answer this. In our work GPU Forecasters, we study language models as selective surrogates for GPU kernel optimization. 1️⃣ Off-the-shelf LLMs can forecast how a GPU responds to a candidate kernel with non-trivial accuracy. If we rank candidates by these predictions and measure only the top 10% on a GPU, the fastest kernel we find is within 20% of the best in the pool. 2️⃣ We want LLMs to not just be accurate but also calibrated, so that we can use their uncertainty for selective prediction: during search, we should trust only confident forecasts and verify less confident forecasts by sending them to the GPU. 3️⃣ We train an open-weights surrogate (GPT-OSS-20B) with RL to improve both accuracy and calibration. Calibration-shaped rewards improve both confidence reliability and ranking ability, while correctness rewards alone do not. 4️⃣ Inside a real kernel search, the surrogate finds faster kernels than an equal-GPU-budget baseline by considering more candidates per measurement. 5️⃣ We release 12,388 LLM-generated GPU kernels with measured runtimes spanning 118 operations, CUDA and Triton backends, 3 GPU types, taking 400M tokens + 600 GPU-hours to produce. This dataset can be used for analyzing LLM-driven evolutionary program search dynamics, post-training LLMs for kernel code generation, and things we didn’t get a chance to explore, like training reward models! Thread 🧵👇

Can the world model be used as a simulator for planning? We designed scenarios where an agent must choose between two plans (e.g., craft required tools before mining vs. mine immediately). In all cases, simulating rollouts within the learned model produces the same ranking of plans as the ground truth environment. The learned world model can simulate diverse mechanics: stochastic NPC movement, deterministic crafting logic, resource consumption, and multi-step action sequences (30+ steps in length).

We tested the learned model across 23 scenarios covering the full range of game mechanics: resource gathering, tool crafting, building, and combat. OneLife outperforms a strong PoE-World baseline on 16/23 scenarios tested and on both metrics, particularly on the ability to distinguish implausible from plausible future states (MRR of 0.479 vs 0.351). The improvements are distributed across different types of mechanics, indicating the model learns a broad understanding of the environment.

Evaluating a world model on random rollouts may not provide sufficient coverage of rare or important events in an environment. We create a suite of evaluation scenarios covering all the core Crafter mechanics. Each scenario runs a short, scripted policy from an initial state designed to reliably exercise a specific game mechanic or achieve a particular goal. We evaluate learned world models on two criteria: 1️⃣ State Ranking: Can the model distinguish plausible future states from impossible ones? 2️⃣ State Fidelity: How closely do generated states match reality? To create distractor states for ranking, we use mutators, which are programmatic functions that apply semantically meaningful, rule-breaking changes to the true next state. For example, a mutator could change a character’s position to a location they cannot physically reach.

We built Crafter-OO: a re-implementation of the Crafter environment with a fully-exposed, object-oriented state. The environment includes stochastic NPCs, hierarchical crafting mechanics, and survival elements. This results in a testbed where an agent can reverse engineer complex dynamics from limited interaction. The object-oriented state is readable by LLMs and programmatically editable, allowing LLMs to reason about and propose programmatic laws describing observed changes.