Let's learn about CUDA graphs, one of the core concepts in inference engines. The CPU is responsible for running programs and launches kernels, and the GPU for executing them. Every time the CPU hits a GPU op, it pushes a command into a stream, which is a queue of commands the GPU drains in order. Each push costs CPU time: a few µs in the driver, and more like 10-20 µs once PyTorch's dispatch and shape checks are added on top. For big kernels this doesn't matter. A prefill GEMM runs for milliseconds and the CPU stays far ahead. Decode is different. One decode step of an 8B model at batch 1 is a few ms of GPU work spread over hundreds of tiny kernels (norm, qkv, rotary, kv write, attention, o proj, mlp, for every layer). Launching each kernel now takes longer than running it, so the GPU finishes and sits idle waiting for the next one. The workload is launch bound, and in a profiler like Nsight Systems you can see it as gaps between kernels. CUDA graphs fix this by recording the whole step once and replaying it with a single launch. Hundreds of CPU launches become one call. And since the driver knows the full workload upfront, it can submit it with smaller gaps between kernels too. That's why decode in vLLM, SGLang, TensorRT-LLM and llama.cpp runs on graphs.

I'm an avid user of coding agents, but there's something I've been struggling to put into words which I think is encapsulated well by these screenshots. these are two PRs to fix the same bug: the first is a +627 / -291 line change, from Claude, the second is a +3 / -2 line change, from Matt, a human (we closed the former and merged the latter). there's a certain coherence in a human's mental model of the codebase which makes some bugs trivial for humans and still really difficult (or token-expensive) for agents. this goes beyond just better memory of the codebase. humans will build a mental map and a probability matrix for which parts of the codebase are most likely to surface a particular bug, given the codebase's coherence, and start by tackling those before going another layer deeper. without matt's contribution we might well have merged the +627 / -291 PR, in turn losing coherence, and becoming more token-dependent as a result. after you lose coherence it's game over. i've found myself pair programming with agents on tiny PRs after a huge amount of independent research / agent swarms to tackle a given problem, and I think this is pretty effective; you can combine the things that agents are naturally good at (parallelization, verifiable problems) with your human coherence. often this results in 15-20 PRs, not all of which get merged.

Local AI 202: Get Massive Speed-ups with Speculative Decoding xitter.cf/i/broadcasts/1NGaroZve…

Using AI to write for you - start to finish - is really just you robbing yourself of one of life’s greatest joys: to think through a problem and figure out what you think about it, and in the process becoming passionate enough to take on the new challenge of solving it.

Learn about how to get multipliers in local LLM speed-ups using speculative decoding in this livestream with @mervenoyann, @MaartenGr and @_EldarKurtic youtube.com/watch?v=H4azmWcq…

I love 3 idiots far before I entered college. At the age of 10, I watched it for the first time. And even my laptop at that time do not have any sound, I only watched the video without audio, I loved it deeply

今天和一位刚开始参与 SGLang Omni 社区的开发者吃饭，他跟我说了一个我觉得非常真实的焦虑： SGLang Omni 发展得太快了。 今天看到 full-duplex 的 PR，明天看到 ASR，后天又是 TTS。每一个方向都觉得应该学，但还没来得及学深入，别人已经又往前走了很多。 最后就会产生一种感觉： 是不是大家都已经懂很多了，只有我跟不上？ 还有另一种焦虑更直接： 感觉 SGLang 这种项目都是一群做 AI、做 systems 的人才能参与，普通 SDE 根本进不去。 我其实特别想说：完全不是。 我自己也不是科班 system 出身。本科的时候，我的 systems 基础也绝对算不上好。 很多 SDE 会觉得，AI infra 应该是那些做模型、做 systems 的人才能搞。 但我以前也会反过来觉得： infra 是不是只有那些从本科开始就天天写 kernel、compiler、distributed systems 的人才能搞？ 后来发现根本不是这么回事。 Infra software engineering 最重要的东西之一，从来不是你一开始知道多少，而是你是不是一个负责任的工程师。 你愿不愿意把代码读懂。 愿不愿意问清楚一个模块为什么存在。 愿不愿意 benchmark。 愿不愿意验证自己的假设。 出了 regression，愿不愿意追到底。 不知道的东西，愿不愿意承认不知道，然后把它学会。 在 AI 时代，这件事情反而变得更加极端。 一个负责任的人，可以用 AI 把自己的学习速度和工程能力放大很多倍。 一个原来不懂某个 subsystem 的 SDE，可以让 Agent 帮他读代码、解释调用链、找相关 PR、搭实验、写第一版实现，再由自己一点一点验证。 但 AI 同样也可以把一个不负责任的人放大很多倍。 你自己都不知道代码在干什么，只因为 Agent 说这个优化有效就 merge；benchmark 没看懂，correctness 没验证，出了问题也不知道从哪里 debug。 这种时候，Agent 不是让工程变快了。 只是让破坏发生得更快。 所以我越来越觉得： AI 放大的首先不是能力，而是人的工程习惯。 好的 engineer 会被放大。 坏的 engineering practice 也会被放大。 如果你现在想进入 AI systems，我反而不建议一上来试图理解整个 SGLang Omni。 那几乎一定会焦虑。 你不需要同时搞懂 full-duplex、ASR、TTS、runtime、scheduler、kernel 和所有模型。 找一个足够小、但足够完整的 vertical slice，然后真正 own 它。 比如我现在非常推荐一个练手方向： 去优化 MLX 上的 SGLang Omni，拿 Qwen3 ASR 做一条完整的优化链路。 不要只改一个孤立的小函数。 把模型跑起来。 看输入怎么进来。 看数据经过哪些组件。 profile bottleneck。 理解为什么慢。 做一个改动。 benchmark。 验证 correctness。 发现自己第一次的理解可能是错的。 再回去读代码。 最后把 PR 写到一个别人敢 merge 的程度。 一个这样的任务做完，你学到的远远不只是怎么让一个模型快 10%。 你会亲自走一遍一个真实 AI inference system 的完整工程生命周期。 这种经验比你坐在那里试图把整个 repo 从第一行读到最后一行有效得多。 另外也非常推荐去看 Andrej Karpathy 那些从零开始搭 neural network / GPT 的内容。 我很喜欢这种学习方式： 不要因为一个系统看起来复杂，就先花半年把所有 prerequisite 学完。 把一个小东西真的 build 出来。 然后当它坏掉的时候，你自然会知道自己下一步应该学什么。 所以如果你最近看 SGLang Omni 的 PR 看得很焦虑，我反而建议你别追了。 项目永远会比你学得快。 选一条链路，慢下来，把它真正做懂。 等你完整地做过一遍以后，你会突然发现： 那些之前看起来只有 system expert 才能碰的东西，其实也不过是一层一层需要被理解、验证和负责的代码。 欢迎来一起折腾。

I am sure you all know this by now that @mervenoyann is an excellent technical writer. Give this a read. Short crisp and to the point! llama.app/docs/prefill-vs-de…