
PhD Student @BerkeleySky | Prev. @UWSyFI @sjtu1896 | https://xitter.cf/t.co/ryX6LxgDpV | Systems & AI
Joined August 2019
- Tweets59
- Following320
- Followers118
- Likes358
Super cool work! Congrats @MangQiuyang 🥳
Yi Pan retweeted
This project has been a lifesaver tbh. It's not only high fidelity, but its agent-first design and modularity makes it super easy to iterate on new ideas! I've been using it for some RL work (coming soon 👀 ). I've been seen saying to follow @KanZhu854772 and @UWSyFi!
🔥 Optimizing LLM serving requires repeated experiments, and running them on real hardware is slow and costly.
Performance depends on how models, kernels, hardware, and serving policies interact. As models and workloads change, engineers must revisit their configuration choices.
🚀 Introducing ServingStudio: an integrated workbench for simulating, analyzing, and optimizing LLM serving systems.
🧵 More in the thread below. (1/9)
Yi Pan retweeted
GPU time is too expensive for blind trial and error.
Introducing ServingStudio: simulate LLM serving configurations, use an agent to find and implement promising optimizations, then validate on real hardware.
🔥 Optimizing LLM serving requires repeated experiments, and running them on real hardware is slow and costly.
Performance depends on how models, kernels, hardware, and serving policies interact. As models and workloads change, engineers must revisit their configuration choices.
🚀 Introducing ServingStudio: an integrated workbench for simulating, analyzing, and optimizing LLM serving systems.
🧵 More in the thread below. (1/9)
My biggest frustration in vibe-coding AI Infra is always the feedback latency. Real runs are slow, so I either let the agent thrash through expensive experiments or micromanage it. ServingStudio solves that for me: simulation predicts, the agent implements, hardware confirms.
🔥 Optimizing LLM serving requires repeated experiments, and running them on real hardware is slow and costly.
Performance depends on how models, kernels, hardware, and serving policies interact. As models and workloads change, engineers must revisit their configuration choices.
🚀 Introducing ServingStudio: an integrated workbench for simulating, analyzing, and optimizing LLM serving systems.
🧵 More in the thread below. (1/9)
Turns out your model might be overpaying for the premium harness while the simpler one is delivering nearly the same result with lower cost 😂💸
Does your Claude model really need Claude Code…? 🤔
We evaluate 7 models on Claude Code, Codex, and Pi. Three surprising findings emerge:
1️⃣Harness choice has little effect on task success rate, but can significantly affect the cost
2️⃣A simple harness can be competitive
3️⃣The native harness isn’t always the best.
Millions of people are using coding agents, but the impact of harness choice remains unclear.
(1/n) More details in the thread. 🧵
Great work! Love the key idea that the spec co-evolves with the code to make sure speedups don't come from reward hacking 🫠🚀
Agents let us build systems for different workloads and requirements. But… can we trust what they build?
We release 🌟SkySynth🌟: an engine for synthesizing high-performance, just-in-time (JIT) systems we can trust, by co-evolving formal proofs and tests alongside the code.
Results:
💿 KV stores up to 2.3× faster than Redis and FASTER + formally verified stores with 2.9× Claude Code's pass rate
🚏 Model routers up to 48% cheaper than a general router
⚡ Specialized inference engine with 2.2× the throughput of vLLM/SGLang
🧵👇
Also reminds me of VibeServe from my old lab @UWSyFi which showed agents can build the system, and this is how you trust it!
Yi Pan retweeted
Qwen3.8-Flash-Next now runs on a single RTX 5090 at 68.3 tok/s — no extreme quantization, no speculative decoding.
· Production-level checkpoint: GB300-validated NVFP4 checkpoint from RadixArk
· 63GB host RAM — less than what 1-bit quants of this model need
· The 51GB n-gram table? It lives on your NVMe, at ~0.5% throughput cost
In the video: one prompt → a playable minecraft-style world, then FreeToken on my 5090 desktop fixing a real FreeToken bug in 10.5 minutes.
Meet FreeToken 🧵
Amazing work! Congrats @HaochengXiUCB
Open-source video generation is now faster than playback without compromising quality.
Introducing Video Delta Net (VDN): hybrid attention for live text-to-video with near-lossless quality.
VDN accelerates Minimax-H3 by 75 - 90 x, generating 14 seconds of 768p video in 11 seconds on 8× NVIDIA B200 GPUs.
Checkpoints + training/inference code + Technical Blog ⬇️
(1/6)
Amazing work!
Your gaming PC can now serve frontier models at interactive speed using official checkpoints without extreme quantization!
Qwen3.6 35B → 8GB RTX 4060 laptop @ 39 tok/s
DeepSeek-V4-Flash 284B → RTX 5090 desktop @ 22-25 tok/s
GLM-5.2 753B → RTX PRO 6000 workstation @ 15 tok/s
Run your claude code or codex now with frontier model for $0
Meet FreeToken 🧵
exciting to see M* from @UWSyFi get adopted. congrats to the team!
Yi Pan retweeted
If you want an in-depth analysis of how your coding agents work when you use them, based on thousands of traces from *real* workloads, check out TraceLab!
Paper: arxiv.org/abs/2606.30560
Website: tracelab.cs.washington.edu/
During an agentic session, a quick “hi” after a coffee break can cost a full dollar.
But it’s not because of the “hi.” It’s the million tokens of context you’re repaying for the moment your cache goes cold.
Here's the caching math from @evan_a_frick:
0:00 Why you're billed for last turn's context, not just this one
0:17 Cache hits: ~10% of full price
0:41 Agentic AI = way more back-and-forth than chat
1:16 Paying full price for context even on a 1-token tool call
1:40 Cache hit vs. cache miss
2:13 The $1 "hi": walk away for an hour, come back to a cold cache
2:59 Cost grows by the square, even with caching
3:28 How context compaction helps
4:03 Why Claude Code/Codex may compact at ~200-300K, not the full 1M window
🔧Coding agents spend a surprising amount of time in the Bash.
In our Claude and Codex traces from TraceLab, Bash accounts for ~56.9% of tool invocations and 56.6% of total tool time.
But what are agents actually running?
We analyze 1.3M executable invocations to uncover command patterns, composition, and runtime behavior.
Thread 🧵(1/n)
The hardest LLM serving bugs are the ones where it looked like everything worked, but accuracy quietly regressed.
Thrilled to share our latest work, Ekka, at ICML 2026! Ekka is an automated system for diagnosing silent errors in LLM serving frameworks. On real-world silent errors from vLLM/SGLang, Ekka reaches 80% pass@1 / 88% pass@5 diagnosis accuracy, improving over SOTA coding-agent baselines by 24–34%, and diagnosing 4 new bugs confirmed by developers. If you’re at ICML 2026, come say hi at Poster Session 5, Hall A #2405, on Wednesday, July 8, 5:00–6:45 PM KST.
Joint work between @UWSyfi and @awscloud with @MichaelGu341332, Zhen Zhang, Shaowei Zhu, Xinwei Fu, Jun Wu, Yida Wang, and @bariskasikci.
Paper 📗: arxiv.org/abs/2606.04594
ICML Page🌐: icml.cc/virtual/2026/poster/…
Blog: syfi.cs.washington.edu/blog/…
Yi Pan retweeted
Every team shipping a coding agent — Claude Code, Codex, Cursor — is really running a serving-systems problem. The "tech behind the tech" is the LLM-serving stack underneath, and until now nobody had real data on what that workload looks like.
New arXiv (2606.30560) from @bariskasikci's SyFI lab (@UWSyFi, @uwcse) is the first large cross-provider trace of real coding-agent use: ~4,300 sessions, 350K LLM steps, 430K tool calls, 43 developers, 8 months, Claude Code + Codex.
It breaks the intuition that agents mean long generations. The median step replays ~119K context tokens to emit just ~214 output tokens — two orders of magnitude more reading than writing. So the bill is the context, not the answer: prefix tokens are 59.5% of total cost.
Tool calls are brutally long-tailed: 80+ tools, but the top 3 are 80%+ of calls, and the 4% of calls that run >1 min eat 85% of all tool time.
And the prefix cache everyone leans on? 95.7% hit rate — yet misses cluster right after a human pauses to think, amplifying prefill 3.8x. Those human-gap misses alone are ~46% of fresh tokens and ~13% of spend.
For technical leaders: your agent's cost and latency live in the loop, the replayed context, and the idle gaps — not raw token generation. Tune tool-call overhead, append-length-aware prefill, and KV-cache eviction around human gaps before you scale the fleet.
Yi Pan retweeted
To make serving coding agents more efficient, we need better hills to climb than traces from synthetic benchmarks like SWE-Bench. This work led by @KanZhu854772 is critical in curating a realistic workload for the ML Sys community to climb! (plus you can analyze your own usage!)
🔥 Coding agents have become one of the hottest LLM workloads. But serving them looks nothing like serving a chatbot: 294× more input than output, hundreds of thousands of tool calls, and extremely long-tailed latency.
🚀 We are releasing the SyFI Coding Trace: ~4,300 real-world coding-agent sessions from our daily use, plus TraceLab, an open-source pipeline to collect, sanitize, analyze, and replay your own traces.
More in the thread below 🧵👇 (1/n)
Yi Pan retweeted
Checkout this great statistic and analysis on modern agentic inference workloads!
🔥 Coding agents have become one of the hottest LLM workloads. But serving them looks nothing like serving a chatbot: 294× more input than output, hundreds of thousands of tool calls, and extremely long-tailed latency.
🚀 We are releasing the SyFI Coding Trace: ~4,300 real-world coding-agent sessions from our daily use, plus TraceLab, an open-source pipeline to collect, sanitize, analyze, and replay your own traces.
More in the thread below 🧵👇 (1/n)
Check out our latest work on coding agent serving led by @serendipity_zk ! If you want to analyze characteristics of coding agents, collect your own vibe coding data, or use traces to optimize serving, give it a spin!
🔥 Coding agents have become one of the hottest LLM workloads. But serving them looks nothing like serving a chatbot: 294× more input than output, hundreds of thousands of tool calls, and extremely long-tailed latency.
🚀 We are releasing the SyFI Coding Trace: ~4,300 real-world coding-agent sessions from our daily use, plus TraceLab, an open-source pipeline to collect, sanitize, analyze, and replay your own traces.
More in the thread below 🧵👇 (1/n)
Check out our recent work!
New distributed training strategies should not require new distributed runtimes.
Introducing Piper: a programmable PyTorch training system for deploying complex training strategies by separating model placement and GPU scheduling from model code.
📄 arxiv.org/abs/2606.11169
Huge congrats to the team!!!
Super stoked that UW SyFI (syfi.cs.washington.edu/) members won a number of prizes at the MLSys'26 competition, NVIDIA Track. Hugre congrats to @KeisukeKamahori , @sudopowr , Yile Gu, Wei Shen, Steven Gao! Thanks to @nvidia , @modal , and the Flashinfer team for the support.
1st place in the GDN Track — Full-Agent Approach
2nd place in the GDN Track — Agent-Assisted Approach
3rd place in the DSA Track — Full-Agent Approach























