Incoming Assistant Professor @HKUSTGuangzhou, PhD @sgSMU
Joined November 2016
- Tweets134
- Following318
- Followers179
- Likes423
Our earlier paper “Rethinking the Value of Agent-Generated Tests...” was rejected multiple times by software engineering venues for its counter-intuitive findings. Now it has been independently rediscovered and is getting the attention it didn’t get in paper review 😃
ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence
across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all
i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all
i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet
also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
Zhensu Sun retweeted
ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence
across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all
i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all
i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet
also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
Benchmarks leak and saturate. So we let coding agents test each other: one hides a subtle bug in a real repo, the other has to find and fix it. We found that Claude Opus 4.8 hides the hardest bugs, but GPT-5.5 is the best finder and fixer.
arxiv.org/abs/2610.12569
In our ICSE'26 paper, we quantified the cost of code formatting in source code and showed empirically that removing this optional formatting does not hurt model performance. A year after the paper's release, it's great to see this finding appears in frontier models 😃
astra has a code quality problem.
let me be a little more precise. astra *can* write good code. but when it infers its in a situation where no one will ever actually look at the code, it does not write it with human readers in mind. it also doesn't write it with *long term maintenance* in mind. it seems to write compressed machineslop that solves the problem at hand in as few tokens as possible while still staying understandable to *astra*.
astra has an reward hacking problem.
i imagine this is what happens when enough software RL environments test strictly functionality or outcomes rather than having any monitoring or reward signal on code quality. maybe in past iterations there wasn't so much optimization pressure, so eg Sol would still engage its "write good code" module even if it thought no one would read it, because that was the only thing it had really learned to do well? but Astra has been through enough iterations in tiny boxes where another machine is judging it that this seems natural.
i don't see these kinds of issues occur yet when working on existing codebases. but keep an eye out. and especially on greenfield projects, make sure to actually look at the code. even if you explicitly tell it you're intending to continue building long-term, it has a *strong* pull to... this
Our recent work introduces a new axis for reducing agent evaluation cost. Specifically, we stop each task early once its outcome becomes predictable. This early-stop strategy fits benchmarks very well, as we can utilize their answers and previous submissions for predicting.
Cut agent evaluation costs by up to 44%
EarlyEval trains LightGBM classifiers to predict an agent's final outcome from its intermediate behavior, halting runs early when success or failure is already clear. Across SWE-bench, TerminalBench, and Toolathlon it cuts 13-26% of steps with minimal accuracy loss.
Our latest work :) Hopefully it can help the design of future performance-optimization benchmarks.
Researchers replayed 740 "official" coding-agent benchmark patches on different cloud machines. only 11 out of 140 kept their validity signal.
a team from singapore management university and shanghai jiao tong university audited the three benchmarks everyone cites to measure whether coding agents can optimize code. gso, swe-perf, and swe-fficiency. they took every official reference patch and ran it across four google cloud machine types, three rounds each.
the results are rough. swe-perf kept only 11 out of 140 tasks valid. gso kept 39 out of 102. swe-fficiency held up best at 411 out of 498 but still lost 87 tasks. these are the reference patches the benchmarks themselves say are correct. swap the machine and the optimization signal disappears.
it gets worse. the same eight coding agents rank differently depending on which benchmark you look at. nine out of twenty-eight head-to-head comparisons flip between gso and swe-fficiency. one model jumped five positions. and swe-fficiency's scoring formula lets the worst ten tasks carry 58 to 82 percent of a submission's entire score weight. one bad task can outweigh hundreds of good ones.
the deeper finding is that 384 out of 450 valid tasks are already matched or beaten by at least one public submission. the benchmarks are running out of room to separate strong agents from weak ones.
swe-perf's core problem is that its reference patches cluster near zero runtime change. the median improvement is negative 0.03 percent. at that scale, normal machine noise is bigger than the optimization itself. the benchmark is measuring hardware jitter, not agent capability.
we keep treating leaderboard scores as hard evidence that coding agents are improving. this paper is a reminder that the ruler we're using bends depending on which shelf you put it on.
paper link in the reply.
The conclusion of this paper is very interesting. Images may be a more friendly representation for AI to understand source code, since it can be easily compressed to reduce cost. 😃
today most interesting paper: CodeOCR
This work provides a good explanation of how code indentation and highlighting are designed to serve the human eye.
arxiv.org/pdf/2602.01785
How could two ICSE reviewers think my paper is novel while the remaining one think the paper is incremental? It doesn't make sense😮💨
Zhensu Sun retweeted
Stripping code formatting cuts LLM token cost without hurting accuracy.
Average input tokens drop by 24.5%, with output quality basically unchanged.
The core issue is simple, indentation, spaces, and newlines help humans read but they inflate tokens that models pay to process.
They remove only cosmetic formatting while keeping program meaning identical, checked by matching the abstract syntax tree of the code.
They test Fill in the Middle code completion, where a model fills a missing block, across Java, C++, C#, and Python.
Performance stays stable on unformatted input, big models barely move, smaller ones wobble a bit, Python sees less savings because its layout is part of the language.
One surprise, models still print nicely formatted code even when given smashed input, so output token savings are small.
To fix that, 2 cheap tactics work, explicit prompts that say output without formatting, and light fine tuning on unformatted samples.
With clear instructions or tiny training, output length shrinks by 25% to 36% while pass rate on the first try holds.
They also ship a tool that strips formatting before inference then restores it after, so humans read clean code while the model pays less.
----
Paper – arxiv. org/abs/2508.13666
Paper Title: "The Hidden Cost of Readability: How Code Formatting Silently Consumes Your LLM Budget"
Want to save your LLM budget without sacrificing performance? Here's a useful trick: removing non-essential code formatting, like indentations, newlines, and extra whitespaces, cuts input tokens by an average of 24.5%!
Check out our full study: arxiv.org/abs/2508.13666
Zhensu Sun retweeted
🚨 Big Announcement! 🚨 We’re thrilled to welcome two distinguished keynote speakers to #FORGE2025!
✨ Prem Devanbu @devanbu (@UCDavis Professor)
🔗 cs.ucdavis.edu/~devanbu/
✨ Graham Neubig @gneubig (@CarnegieMellon Associate Professor )
🔗 phontron.com/
I'll ride a dog to lab in a near future.
Unitree B2-W Talent Awakening! 🥳
One year after mass production kicked off, Unitree’s B2-W Industrial Wheel has been upgraded with more exciting capabilities.
Please always use robots safely and friendly.
#Unitree #Quadruped #Robotdog #Parkour #EmbodiedAI #IndustrialRobot #InspectionRobot #IntelligentRobot #FoundationModels #LeggedRobot #WheeledLegs
Zhensu Sun retweeted
🎉 Exciting News! 🎉
We are thrilled to announce that ACM SIGSOFT has officially upgraded FORGE from an ICSE Special Event to an ICSE Co-Located Conference! 🚀 We can’t wait to see your submissions for FORGE 2025!
See more below👇
#FORGE #FORGE2025 @ICSEconf
Zhensu Sun retweeted
AI is not making any progress"? Look closer. 🙄 GPT-4 level models got 240x cheaper in just 2 years! AI progress isn't linear and is just about bigger models.
BERT -> DistilBERT
Llama 2 70B -> Llama 3 8B
GPT-4 -> GPT-4o-mini
Llama 3 405B → Llama 4 70B?? 🤔
Models get bigger, then smaller but equally powerful. It's a cycle of innovation. Today's quality per $ is the most expensive we'll see.
Making it cheaper will lead to more people using, learning, and building with AI, which might unlock more potential and “goodput” for everyone than yet another Foundation Model!
AI's real progress: Getting into more hands.🤗
[Image credits: @davidtsong]
