Researcher @Amazon | ML Ph.D. @GeorgiaTech | ex-researcher @ai4opt | Gen AI, Agentic AI, Optimization
San Diego, CA
Joined June 2019
- Tweets41
- Following140
- Followers66
- Likes116
Wenbo Chen retweeted
Introducing Robo Use: API to physical worlds for agents
One interface, 32 embodiments, 100+ scenarios, and 11 agents and models
We aggregated all academic simulator scenarios and ran 2500+ trials with initial results.
Preview them or try it yourself 🧵
Wenbo Chen retweeted
FrontierPhysics update #3
We're wrapping up the v0.1 leaderboard.
From initial runs on 56 real physics research tasks, the strongest frontier model passes only 22% of its runs.
How we got a number we trust 🧵
Wenbo Chen retweeted
We've spent decades instrumenting programs: LLVM, static & dynamic analysis.
LLM agents now act inside complex eval infrastructure—reading and editing files, running shell commands, calling tools, browsing the web, submitting artifacts, and collecting rewards. Every one of those actions can shape the final score. They deserve the same treatment.
Introducing "BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure".
📄 arxiv.org/abs/2609.11028
We instrument the eval infrastructure itself: static taint analysis exposes reward-hacking paths before any agent runs, and runtime evidence shows whether a concrete run actually used one.
Wenbo Chen retweeted
It's well known that agents hack benchmark rewards.
The usual response is a patch for each task that gets exploited.
In a study of 456 adjudicated trajectories from more than 31,000 public agent runs, 69% contained at least one reward-hacking episode.
Most of the exploits appeared mid-run after legitimate work.
BenchShield models each evaluation as a finite set of reward-relevant events.
A static taint analysis finds hack paths from the task package before any run. A runtime pass then uses evidence from the benchmark infrastructure to decide whether the agent actually used one.
On Terminal-Bench 3, SkillsBench and ClawsBench, the static pass recovers 77 to 100% of exploit chains, against 23 to 94% for an agentic scanner, at up to 65% lower cost. Runtime detection reaches 96% accuracy, against 36% for an LLM reading the transcript.
Paper: academy.dair.ai/papers/bench…
Wenbo Chen retweeted
Super honored to cowork with so many amazing physicist and AI researchers!
As I read through all the tasks I feel I am reading AI-native papers and have learnt so much new physics knowledge from all kinds of domains❤️
Join us if you believe in AI4S!🚀
FrontierPhysics bi-weekly update #1
1st benchmark for e2e frontier physics research
Within 2 weeks
- 10+ nature authors contributed
- 1.6k+ community members, 400 new
- 200+ contributor signup
- 60+ task submissions
- physics professors from UCB, UCSD, UMich
more in 🧵
Wenbo Chen retweeted
Excited news and great to see SkillsBench cited!
Introducing Muse Glimmer, an open-weight 30B-parameter model optimized for local, always-on agent workflows.
Muse Glimmer delivers strong performance on key agentic use cases and benchmarks compared with leading models in its size category, and is designed to run entirely on consumer hardware like a Mac or PCs with performant GPUs.
In keeping with our long tradition of sharing fundamental AI research, we’re releasing model weights under a permissive Apache 2.0 license.
🧵👇
Wenbo Chen retweeted
Introducing FrontierPhysics: benchmark for e2e frontier physics research
We built SkillsBench and got 180+ citations in 6mo. People have been asking next project to contribute to
Led by @bingran_bry - Physics PhD @UCBerkeley, Nature author, advised by Prof. Haeffner and more 🧵
Wenbo Chen retweeted
SkillsBench used by @github copilot team for harness engineering
Predicting this will become a default part of harness evals
Had a lot of fun in the journey! Kudos to the team!
A big pain point in using AI benchmarks is encountering errors after its first release. Today, we're releasing SkillsBench 1.1, the first benchmark for how well AI agents use skills, now audited end to end and verified error-free. Prof. @dawnsongtweets joins 1.1 as advising author.
We worked through every task with several frontier labs to eliminate the errors in the previous version. We also added new tasks, moved the ones with external dependencies into a separate set so the core suite runs clean, and expanded coverage to more models.
Capability is climbing fast. The best with-skills resolution rate rose from ~36% (Claude Sonnet 4.5, Sep 2025) to 67% (GPT-5.5, May 2026), about +1.9 points per month. The frontier is hill-climbing SkillsBench fast.
The right skills still matter. Across the fleet, curated skills lift resolution rate by +16.6 points on average (33.9% → 50.5%), and by as much as +25.7 points for a single model. The top configuration is GPT-5.5 on OpenHands at 67.3%.
By popular demand (thx Nate @cursor_ai), we're now tracking skills invocation: how often an agent actually uses the skills it's given. Recent flagship configurations invoke them 90–99% of the time (Codex 99%, OpenHands + GPT-5.5 92%, Gemini CLI 90%), versus roughly 50% for older setups.
Also new in 1.1: @OpenHands joins as a fourth harness, alongside Claude Code, Codex, and Gemini CLI; a rebuilt leaderboard with refined categories, subdomain skill rankings, and Skill Lift; and native task . md on BenchFlow, with multi-scene environments and rollout branching. We also partnered with @k_dense_ai to add scientific skills to some science tasks.
One implication for deployment: skills can substitute for scale. GLM 5.1 with skills (58.4%) outperforms Opus 4.8 without (45.7%). A smaller model with the right procedural knowledge can beat a larger one running without it.
Huge thanks to @nick_kango @ivanleomk @kaggle @GoogleDeepMind for hosting a launch event with us. Thanks for everyone who's come on May 27!
Also thanks to our partners @gneubig @OpenHandsDev @ivanburazin @daytonaio @jackminong @johannes_hage @PrimeIntellect @TimothyKassis @k_dense_ai for providing support in credits, compute, and skills.
SkillsBench live leaderboard will also come to @ValsAI. Many people have told us they use SkillsBench as an index to measure models' agentic capability over diverse and high GDP value domains. Great work on Valkyrie as well! @ Jarett @nikilravi @langstonnashold @RayanKrishnan
SkillsBench is fully open-source. Explore the leaderboard and tasks, read the docs, or contribute your own skill set or harness and join the leaderboard. 🧵
Wenbo Chen retweeted
🎉 Excited to share: LSFLOW is accepted as a Spotlight at #ICML2026 (top 2%)!
The first flow/diffusion policy framework for RL with combinatorial actions. 🚀
📄 arxiv.org/abs/2601.22211
💻 github.com/anagha-satish/com…
🧵👇
Wenbo Chen retweeted
competitors dont matter
rejections dont matter
status dont matter
people matter, build matter, sales matter
almost 2yrs building a company. these are the most rewarding parts.
Wenbo Chen retweeted
What a wonderful experience co-organizing Agent Skills '26 workshop at @CAISconf. We had a full house!
Huge thanks to our speakers @dawnsongtweets @ManlingLi_ @gneubig @Yushun_Dong @kanavg1 @ysu_nlp and panelists @obra @robennals @ysu_nlp for the talks and discussion that made the day,
and to the entire organizing team who made it all happen 🙏.
Great job! @xdotli @Yimin1010
after a few days of work and grind, skillsbench is finally live on AgentBeats before the Spring 4! Kudos to my brother @Yimin1010 and thanks @dawnsongtweets for the projects - learned a lot during working around with the infra. really awesome work
Wenbo Chen retweeted
🎓 Thrilled to share: I'll be joining The University of Hong Kong, School of Computing and Data Science as a Tenure-track Assistant Professor in Fall 2026! We are building GenAI agents (diffusion / LLMs + RL) for real societal impact.
📣 Recruiting PhDs, a postdoc, and RAs — email [email protected] 🚀
#AcademicJobs #GenerativeAI #AgenticAI #LLM
Wenbo Chen retweeted
OpenReview is now public for the @CAISconf Agent Skills workshop
103 submissions, 45 posters, 6 orals
Absolutely incredible results for a workshop at an inaugural conference. Kudos to everyone on the team 🫡
sponsors from @k_dense_ai (largest scientific skills repo) 👏
Wenbo Chen retweeted
We will share
* SkillsBench 1.0, recipes on creating successful evals
* RL env creation, qa, running at scale, against different harnesses
at SkillsBench 1.0 launch party
luma.com/deepmind-634c cohosted with @ivanleomk @GoogleDeepMind
Spaces are limited, register!
Introducing @harvey LAB in benchflow-ai/benchmarks
Skills have significantly increased agents deployment in diverse domains outside of coding and more complex environments outside of terminal.
Kudos to Harvey for an amazing open benchmark that demonstrate this 👇🧵
Wenbo Chen retweeted
Hosting the SkillsBench 1.0 launch party with @ivanleomk, @nick_kango with @KernaLabs, @kaggle, and @benchflow_ai
We will release the 1.0 version of the dataset, how we made it, and other secret releases.
Link: luma.com/deepmind-634c
Wenbo Chen retweeted
We got more papers than we expected at our CAIS Agent Skills, so we are 1) extending deadlines to May 4 AOE and 2) recruiting more reviewers
Link: agentskills-workshop.org
Wenbo Chen retweeted
SkillsBench is now cited by HY-3 model card. Congrats to @TencentHunyuan on the launch and kudos to the SkillsBench team / community!
We've made a lot of improvements to the tasks, codebase, tooling in the past month, based on feedbacks from users and lab partners. We will also share updated leaderboard with more models and agent harnesses soon, stay tuned!
