Patronus AI retweeted
Can @thinkymachines new Inkling-2-Small not just play a video game level, but play and refine it many times over and learn to speedrun it?
@akkikiki , myself and other co-authors from @UW and @PatronusAI ran one of the setups from our SpeedRunBench preprint from Sep this year to observe exactly that (arxiv.org/abs/2610.08076).
Thanks to the @thinkymachines and @tinkerapi team for credits that enabled this study!
Our benchmark measures whether agents can discover and then refine offline a fast enough policy through a level as they play more and more attempts, using only a single score after each attempt to adjust their policy.
By design, we don't refine policy amidst episodes, or have multimodal frame understanding ... so our benchmark is more a read on planning, exploration, autoresearch etc than online videogame playing per se..
We used the "offline scratch" setting on SuperTux (SuperMario style game, Tux in place of Mario) level 1-1.
Results:
1. Inkling-2-Small (256K, thinking effort 0.5) is an adept speedrunner, learning to clear the level after 1,988 frames, second only to GLM 5.3 (1,961 frames) in the open source family models we benchmarked ..
2. This put Inkling-2 ahead of Gemini 3.7 Flash, GPT-5.6-sol, Claude Opus 5, and DeepSeek-V4-Flash. See the attached bar graph for the final frames
3. Inkling-2-Small dramatically improves over its predecessor Inkling-Small which never learnt to fully clear the level, let alone improving at speedrunning it.
Note for scale that Human world record iterated over the years for speedrunning this game is 1,619 frames...
Inkling 2 Small’s weights are now available for download on HuggingFace under an Apache 2.0 license.
Make Inkling your own!
huggingface.co/thinkingmachi…
Patronus AI retweeted
RSI is the next frontier in research. But how do we achieve this?
I wrote about the role of data in RSI. Models need to get better at training data attribution, ecologically valid generations, and measuring their own progress. It’s evals all the way down.
Patronus AI retweeted
Synthetic data is often dismissed as AI slop. I disagree.
I wrote about how we should approach synthetic data for mid-training and RL, where it currently falls short, and how human expertise can shape far more tasks than experts could write themselves.
Patronus AI retweeted
True story: our original company name when we were in stealth was Zeno AI, not Patronus AI.
Back in 2023, Rebecca and I started the company on the idea that AI evaluation was like Zeno’s Paradox. Every time models reached the bar we had set, the frontier of what we needed to evaluate moved further ahead.
Fast forward to 2026, I still think Zeno's Paradox is the most accurate way to describe AI progress. Every finish line we draw for AI moves the moment it reaches it.
In the original Zeno's Paradox, every time Achilles reaches where the tortoise stood, the tortoise has moved a little further on, so he appears to chase forever. Mathematically the paradox does resolve. The shrinking gaps sum to a finite length, and Achilles does pass.
In the AI version of the paradox, the idea is that humans are the tortoise, and AI is Achilles. We start far ahead, carrying millions of years of evolutionary and cultural learning. And over the last few years, Achilles has closed most of the visible distance.
But the reason the AI version is more interesting is that the tortoise keeps speeding up. Each time AI closes a gap, it hands the people defining the frontier a better tool for defining the next one. Mathematicians use the newest models to probe harder open problems. Researchers use them to help design the very evals the next generation of models will struggle against.
Everyone is anxious about the rate of progress in AI right now, but ultimately I think we’re going to land in a pretty optimistic place: AI will enable humans to reach farther than ever.
@danshipper has a nice GIF of this tortoise/Achilles setup, see below.
Patronus AI retweeted
We just hired the creator of Fruit Ninja and Jetpack Joyride.
Excited to share that Duncan Curtis is joining us as SVP of Product and Operations at @PatronusAI. Previously, Duncan was Head of Product at Zoox, SVP at Sama, and Product Lead at Google Play. He’s also the one to blame if you lost time playing Fruit Ninja back in the day…
Designing LLM post-training data is oddly similar to designing games.
A great game has simple rules, but creates a huge space of possible strategies. It has to be challenging without being impossible, and the challenge has to actually reveal skill. You can tell the difference between someone who has memorized a trick and someone who has actually mastered the system.
Great RL tasks work the same way. You’re designing a simulation world with rules, tools, and feedback, and you’re testing whether agents can complete tasks through good decision making and NOT reward hacking.
As models get better, I think agent task design will increasingly look like game design... the same principles that make games compelling can make RL envs powerful.
Patronus AI retweeted
RL coding agents increasingly game rewards by exploiting their semantic and syntactic weaknesses. Can LLMs detect such behaviors from live training rollouts?
We find contrastive cluster analysis is key! 🚀
GPT-5.2 jumps from 45% to 63%. Humans reach 90%
Paper + data 🧵
Patronus AI retweeted
Building reliable agents requires a different tech stack: one that natively supports compound AI systems and evaluates quality along the full trajectory of agent behavior.
We teamed up with @PatronusAI to break down what this stack looks like, from infra and models to debuggers.
Patronus AI retweeted
Today, MLCommons is announcing a new collaboration with contributors from across academia, civil society, and industry to co-develop an open agent reliability evaluation standard to operationalize trust in agentic deployments. 1/3
🔗 mlcommons.org/2025/06/ares-a…
Patronus AI retweeted
To make sure your AI agent is not bullshitting you, you need to evaluate its reasoning...
but to do so automatically, you need an LLM...
🤔so how do you evaluate the trace evaluator?
With TRAIL, which contains:
- a full taxonomy of agent errors and most frequent failure cases,
Patronus AI retweeted
What if you had an Agent tailor-made for debugging and improving your Agent?
I am SUPER EXCITED to publish our newest Weaviate Podcast featuring Anand Kannappan (@anandnk24), Co-Founder of Patronus AI (@PatronusAI)! 🎙️🔥
The Patronus AI team has just launched Percival! Percival is transforming the agent debugging experience with:
> 60+ failure mode detection covering tool calls, planning errors, context explosion, and more 🗺️
> Suggested prompt fixes that close the feedback loop instantly 🛠️
> A memory architecture using episodic and semantic retrieval with Weaviate! 🤝🚀
I think this is a massive paradigm shift for how Machine Learning and AI systems have been developed -- The era of static evaluation is ending! 🤯
Percival represents a fundamental shift toward "agentic supervision" - equally capable AI systems providing oversight for increasingly autonomous workflows.
I hope you find the podcast useful! I learned so much from this conversation with Anand, and as always, more than happy to discuss these ideas further with you!
Patronus AI retweeted
Check out the very cool work from our friends @PatronusAI 🔥 work here!
huggingface.co/spaces/Patron…
Patronus AI retweeted
This agentic debugger solves my biggest problem with agents: finding out what went wrong in a 600-line log.
It scans the whole run and flags 20+ failure types:
- bad tool call
- planning loops
- API issues
No more reading long traces through ten different abstractions.
Patronus AI retweeted
Big for agents. You can now automatically find and fix nearly all bugs in your workflows.
@PatronusAI just released a debugging tool that:
▸ Scans full execution traces
▸ Detects 60+ failure types
▸ Suggests prompt fixes
Works with Langchain, CrewAI, OpenAI SDKs, and more.
Patronus AI retweeted
The Vercel AI Accelerator is back with over $2.7M in credits.
Build the future with credits from Vercel, v0, AWS, and top AI platforms.
Applications are open until May 17.
vercel.com/ai-accelerator
Patronus AI retweeted
Fighting Hallucinations is one of the most important features for a RAG system to have! ⚔️
SUPER excited to share a bit of what we've been cooking up with our friends @PatronusAI! 🚀
> The team at Patronus has created Lynx, a custom, state-of-the-art model for Hallucination Detection!
> On the Weaviate side of the coin, we have engineered the Query Agent to "cite its sources".
> This recipe illustrates how you can connect the `sources` response from the Query Agent to Patronus' Lynx evaluator!
The recipe is linked below, I hope this inspires your trust in responses from the Query Agent!
I also really hope you will check out Patronus AI, incredible team! 🔥
Patronus AI retweeted
We’re proud to see several Lightspeed portfolio companies named to this year’s @CBinsights AI 100 list: @Cartesia, @Chainguard_dev, @EveLegalAI, @Fiddler_AI, Helsing, @NoeticaAI, @PatronusAI, and @Saronic.
Now in its ninth year, the AI 100 celebrates the top emerging AI companies shaping the future. This year’s 100 winners were selected from a pool of 17,000+, with a focus on early-stage players showing strong market traction, investor quality, and talent.
Check out the full list: cbinsights.com/research/repo…
Patronus AI retweeted
We went to AI Demo Night at @vercel in SF! 🌉
A vlog from our very own @IanMcclanan ↓
Shoutouts @gumloop @BytespaceAI @Vapi_AI @PatronusAI
Patronus AI retweeted
📣 Attention NYC developers: Join Timescale for a hands-on AI Hack Night on 4/29 in partnership with @Company and their AI Review Series.
- Build with Timescale, PGAI + other cutting-edge AI tools
- Hear from AI experts from Anthropic and Databricks
- Win prizes (and meet other AI builders)
👉 RSVP now (spots limited): lu.ma/3ifarjri
Patronus AI retweeted
Today I’m excited to introduce the Patronus MCP Server, the first MCP server for agent testing and optimization.
git clone github.com/patronus-ai/patro…
Here’s why I’m excited about this. ⬇️

























