Today we’re releasing LEAPBench, a benchmark for how efficiently models learn from feedback over many rounds. Most evals only score whether a model eventually reaches a good solution. LEAPBench scores how many tries it took to get there. Two systems can reach the same optimum while one needs 7 experiments and the other needs 27. Most benchmarks call that a tie, even though those 20 extra runs cost real time and materials. LEAPBench is 55 multi-turn optimization tasks, 45 in biology and 10 education RCTs, built from 2,719 real experimental observations. In each task, the model proposes parameters, sees the result, and iterates toward an optimum over as many as 30 rounds, e.g. one biology task involves maximizing antibiotic yield by tuning variables like pH, glucose concentration, temperature, and soy flour level.

training data is starting to look like a zero knowledge proof problem. labs have to judge quality without seeing the full dataset or the QC pipeline behind it. vendors proxy quality with multi-rollout pass rates, small-model ablations, and downstream eval gains. but compute and iteration costs explode as environments and trajectories grow more complex. quality has no ceiling, and the best data is often the hardest to capture in a metric or explain in a writeup. huge alpha in making data quality more legible.

two of the highest-leverage roles for operators trying to break into ai right now: spl (strategic projects lead) and em (engagement manager) positions at data companies. day-to-day, they sit between a frontier lab's research team and the experts producing training data, translating between them in real time. one person fusing sales, ops, and technical fluency. these are the most contested hires in ai data right now. six months in and you'll get competing offers from every major data vendor and even research orgs. candidates ghost their own interviews. offers take weeks to close. total comp is through the roof. the implicit deal is 80-hour weeks owning eight- and nine-figure pipelines producing the training data behind frontier models. you will burn out, recover, then run it back until agi. find the team you trust to do this with, and no other role will teach you more or compound your impact on the industry faster.

A startup shut down 9 years ago. Today, its code is in a bidding war. The founder kept everything: Verilog RTL, a custom compiler toolchain, PCB schematics, full commit and PR history. Most production-grade chip design stays locked inside major silicon shops. Two labs have made offers to license the code to build RL training environments. The offers already total seven figures. Check your hard drives.

rl environment companies have a version of the same problem as traditional human data vendors. the know-how is sold instead of compounding. labs receive environments, tasks, and verifiers directly. they can inspect the verifier design, extend the architecture to new domains, and eventually automate the creation of more complex and diverse environments without the original builder. just less obvious. hosted environments could change this. labs get reward signals without ever receiving the docker image. IP stays protected. openreward is an exciting first step towards getting the best environment builders onto a platform where high-quality private environments can be accessed before they saturate, and builders keep what they build.

our team recently completed a study on model metacognition and found that confidence and capability were almost completely uncorrelated across 19 frontier models from different labs. self-certainty as a reward signal assumes confidence correlates with correctness. i wonder whether this holds outside math and code. more to share soon.

expanding what’s deterministically verifiable for RLVR is a methodology bottleneck as much as a tooling one. most verifiers are script-based or llm-as-judge. the opportunity is building reward signals durable enough to open up the domains worth training on.