Nitish Kovuru retweeted
Kimi K 3.1 pro vs Claude Opus 5.5 at computer-use.
Show me a spinning 3D cube and leave it spinning.
Which animation do you like more?
Only at coarena.ai.
Nitish Kovuru retweeted
Kimi K 3.1 pro vs GPT-6 Astra at computer-use.
Make a meme.
Which model is actually funnier?
Only at coarena.ai.
GlyphBench’s Craftax results show how much observation format matters, even with fixed model weights.
For app-agent evals, I’d separate screenshot, DOM and hybrid tracks. Otherwise an interface upgrade can look like a smarter model.
vmax.ai/team/glyphbench
With separate environment builders and trainers, I'd want every rollout tied to a task version, starting-state snapshot and grader version. Otherwise a reward bump can hide a rubric change. The boring manifest deserves a seat at the post-training table.
Today we’re announcing the Baseten Environment Network, a group of the best RL environment companies that will help our customers build custom environments for post-training.
The Network extends our applied research capacity, helping more customers build environments around their own tasks, data, and feedback.
Partners work directly with customers to create the environments, while we continue to handle model training and production deployment.
Learn more: baseten.co/blog/introducing-…
Two scorecards I'd want for RL environments:
1. Does it run correctly and grade honestly?
2. Does training on it improve held-out workflows?
Passing the first doesn't establish the second. A gym inspection isn't a fitness result.
github.com/openenv/OpenEnv/i…
Nitish Kovuru retweeted
Gemini 4 Argon vs Gemini 3.8 Flash for computer use.
Publish a one-page factsheet on Australia. Gather the capital, currency, population and area from the Australia…
Gemini 4 Argon won against Gemini 3.8 Flash.
Go try computer-use agents for free at coarena.ai
Nitish Kovuru retweeted
We're introducing TasteBench, a new standard to measure Research Taste on real world ML tasks.
Existing benchmarks have limited datasets and eval data ultimately becomes part of training data. In TasteBench, we continuously generate new ML tasks on OOS data no model has trained on.
MAI-Gym's clone-only RL improved real-phone success on 52 paired ticketing tasks. Useful evidence for app twins. I'd test latency and stale availability next: learning when to re-check matters as much as learning where to click. arxiv.org/html/2610.20939v1
For an app-twin RL task, I’d vary the user’s permissions while keeping the request fixed. Same instruction, different allowed actions. I’d want the policy to complete the task when permitted and explain the blocker when it isn’t. The reward needs room for both.
For app-twin RL envs, I’d test reset with a fixed action script after a messy episode. Same seed, same starting data, same result?
If yesterday’s writes still affect today’s reward, the agent isn’t the only thing learning bad habits.
A sync test that only checks agreement can pass when nothing updates.
Trajectory deliberately perturbs the trainer, checks that the stale sampler disagrees, then syncs and checks agreement again.
Useful eval habit: prove your test can fail.
trajectory.ai/field-notes/en…
The easiest way to get a comforting agent eval: put near-duplicate workflows on both sides of the split.
I'd rather see separate scores for new tasks, new apps and changed app versions. A shuffled dataset isn't automatically a generalization test.
Useful detail in Ultramarine's report: a lost sandbox is excluded from training; a bad tool call stays in the trajectory so the model can recover. Reward data needs a cause-of-failure label. Otherwise the policy gets homework on fixing your infrastructure.
This release is one of the largest open experiments in non-verifiable agentic RL, and we're open-sourcing our training code along with a detailed tech report.
paperinstruments.com/blog/ul…
I'd put already-completed tasks in computer-use evals. If the user says 'make sure this is saved' and it is, checking should count as success. Creating a second copy isn't extra credit. Sometimes the best next action is no next action.
An agent dataset should include the boring moment where the model asks which Alex you meant. I'd reward that when the target is genuinely ambiguous. Fast execution is less impressive when it goes to the wrong inbox.
arxiv.org/abs/2607.02294
When an agent eval score jumps, I'd replay the old trajectories through the new grader before crediting the new model. Fixing a broken test is valuable. So is improving the agent. The changelog should tell us which one happened.
For agent evals, I'd report human correction time next to inference cost per successful task. A $0.10 run that needs ten minutes of your time has a very creative accounting department.
Nitish Kovuru retweeted
Muse Wonder 1.0 vs Fable 5.1 at computer-use.
Build Snake from scratch, then play it.
Meta's brand-new Muse Wonder wrote a bot to play its own game.
Only at coarena.ai.