Patronus AI retweeted
Can @thinkymachines new Inkling-2-Small not just play a video game level, but play and refine it many times over and learn to speedrun it?
@akkikiki , myself and other co-authors from @UW and @PatronusAI ran one of the setups from our SpeedRunBench preprint from Sep this year to observe exactly that (arxiv.org/abs/2610.08076).
Thanks to the @thinkymachines and @tinkerapi team for credits that enabled this study!
Our benchmark measures whether agents can discover and then refine offline a fast enough policy through a level as they play more and more attempts, using only a single score after each attempt to adjust their policy.
By design, we don't refine policy amidst episodes, or have multimodal frame understanding ... so our benchmark is more a read on planning, exploration, autoresearch etc than online videogame playing per se..
We used the "offline scratch" setting on SuperTux (SuperMario style game, Tux in place of Mario) level 1-1.
Results:
1. Inkling-2-Small (256K, thinking effort 0.5) is an adept speedrunner, learning to clear the level after 1,988 frames, second only to GLM 5.3 (1,961 frames) in the open source family models we benchmarked ..
2. This put Inkling-2 ahead of Gemini 3.7 Flash, GPT-5.6-sol, Claude Opus 5, and DeepSeek-V4-Flash. See the attached bar graph for the final frames
3. Inkling-2-Small dramatically improves over its predecessor Inkling-Small which never learnt to fully clear the level, let alone improving at speedrunning it.
Note for scale that Human world record iterated over the years for speedrunning this game is 1,619 frames...
Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license.
Make Inkling your own!
huggingface.co/thinkingmachi…
Expert data has a selection bias problem.
In an attempt to produce as many tasks as possible, data vendors are chasing the biggest pool of experts they can find. We think this will make models worse at the details of knowledge work. 🧵
Who is the right expert for AI? by @vinanandgov with @devanshuba38471.
patronus.ai/blog/right-exper…
1
11
519
We're hiring expert contractors, researchers, engineers and operators.
patronus.ai/join?utm_source=…
3
458
“Need inspect repo. Grep.”
Last year open models said “Let me start by exploring the repository.” Now several write exclusively in clipped, article-free shorthand.
Across 12 open models × 20 coding tasks, filler words fell 25–50% between releases.
But terser isn’t cheaper. 🧵
Why waste time say lot word when few word do trick?
Few Word Do Trick, by @ArnhavD and @VarunGangal. The full writeup has the per-family charts, the filler rules and the limitations.
patronus.ai/blog/few-word-do…
Arnhav and Varun study how models reason and act. We're hiring researchers and engineers in San Francisco to do behavioral research like this with them.
patronus.ai/join
Today we're open-sourcing GlobeBench, 1,632 tasks that test whether LLMs can simulate RL environments well enough to train agents on.
The top model, GPT-6 Astra, gets 49.6% right. No other model of the 20 we tested reaches 30%. 🧵
We're hiring researchers and engineers to build world models and RL environments with us. patronus.ai/join
2
16
2,280
Coauthor @BannurCharvi is a researcher at Patronus building RL environments and evals. Before Patronus, she was on the research team at @datologyai and did research on synthetic data generation and post-training.
2
318




