Patronus AI (@PatronusAI) header
Patronus AI (@PatronusAI) profile picture

Patronus is a model behavior research lab developing training data and simulated environments for frontier AI. https://xitter.cf/t.co/8X6bVgvCHd

San Francisco, CA
Joined July 2023
Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)Photo by Patronus AI (@PatronusAI)
Patronus AI retweeted
Can @thinkymachines new Inkling-2-Small not just play a video game level, but play and refine it many times over and learn to speedrun it? @akkikiki , myself and other co-authors from @UW and @PatronusAI ran one of the setups from our SpeedRunBench preprint from Sep this year to observe exactly that (arxiv.org/abs/2610.08076). Thanks to the @thinkymachines and @tinkerapi team for credits that enabled this study! Our benchmark measures whether agents can discover and then refine offline a fast enough policy through a level as they play more and more attempts, using only a single score after each attempt to adjust their policy. By design, we don't refine policy amidst episodes, or have multimodal frame understanding ... so our benchmark is more a read on planning, exploration, autoresearch etc than online videogame playing per se.. We used the "offline scratch" setting on SuperTux (SuperMario style game, Tux in place of Mario) level 1-1. Results: 1. Inkling-2-Small (256K, thinking effort 0.5) is an adept speedrunner, learning to clear the level after 1,988 frames, second only to GLM 5.3 (1,961 frames) in the open source family models we benchmarked .. 2. This put Inkling-2 ahead of Gemini 3.7 Flash, GPT-5.6-sol, Claude Opus 5, and DeepSeek-V4-Flash. See the attached bar graph for the final frames 3. Inkling-2-Small dramatically improves over its predecessor Inkling-Small which never learnt to fully clear the level, let alone improving at speedrunning it. Note for scale that Human world record iterated over the years for speedrunning this game is 1,619 frames...
7
17
3
119
8,672
Expert data has a selection bias problem. In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
5
6
6
97
166,767
“Need inspect repo. Grep.” Last year open models said “Let me start by exploring the repository.” Now several write exclusively in clipped, article-free shorthand. Across 12 open models × 20 coding tasks, filler words fell 25–50% between releases. But terser isn’t cheaper. 🧵
10
5
1
41
4,454
Arnhav and Varun study how models reason and act. We're hiring researchers and engineers in San Francisco to do behavioral research like this with them. patronus.ai/join
2
1
5
309
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on. The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. 🧵
34
49
28
736
120,368
Coauthor @BannurCharvi is a researcher at Patronus building RL environments and evals. Before Patronus, she was on the research team at @datologyai and did research on synthetic data generation and post-training.
2
318