hellobert retweeted
Why are we calling every new kernel release a Megakernel. This is a well written CUDA graph engine with persistent kernels and PDL.
The author didn’t even call this a megakernel
github.com/vllm-project/vllm…
BREAKING NEWS🚨: NVIDIA has just achieved 423 tok/s/user 🚀 on the 2.8T Parameter Kimi K3 model via a new upstream vLLM optimization. 1.68x Faster on the Same Hardware!
This was done by NVIDIA CentML's Benjamin, who wrote NVIDIA's new decode-specialized kernels in Cute DSL via aggressive Megakernel fusion. This is a 12,000 LoC Megakernel and is the future of advanced AI-assisted kernel generation. NVIDIA is already working on DeepSeekv4.1 and other models for vLLM too.
Someone should give this NVIDIA 10x engineer a raise for this great new major feature!
hellobert retweeted
zero-to-sglang new chapters are out! 📖
Want to read inference engine source code but don’t know where to start?
We’ve just added two chapters to zero-to-sglang, the open-source AI Infra course from SGLang × Datawhale:
📖 2.1 — mini-sglang: What an Inference Engine Looks Like
📖 2.2 — The Journey of a Request + Code Walkthrough
Find us on GitHub: datawhalechina/zero-to-sglang 🫵
#SGLang #LLMInference #AIInfra #OpenSource #LLM
🤖 Made with AI
hellobert retweeted
We’re sharing how GLM-5.3 helped build and optimize the inference infrastructure serving GLM-5.3-Flash.
The system went from its first successful run to production readiness in less than two weeks, with end-to-end throughput tripling relative to the initial baseline.
The key was dense feedback: local correctness tests, execution traces, microbenchmarks, and end-to-end measurements that enabled targeted hypothesis testing rather than reliance on aggregate performance metrics alone.
z.ai/blog/glm-built-its-infe…
hellobert retweeted
残差连接的改进已经散成八条线（HC、mHC、iHC、Gated Residual、Attention Residuals、xHC、VWN…），这份笔记把它们收进了一篇，做架构的人先收藏 →
github.com/Purshow/Purshow_N…
Residual connections can be considered one of the great milestones that made the “deep” in deep learning possible. In recent years, after improving Transformer architectures in just about every other way, researchers have finally turned their attention to residual connections, giving rise to many interesting modifications. So I put together this brief blog. There may well be omissions or even mistakes, and any corrections or additions are very welcome!
The English version was translated from the original Chinese by GPT-6 Astra. If you can read Chinese, I recommend reading the original notes.
PDF:
[Chinese (CN)](github.com/Purshow/Purshow_N…)
[English (EN)](github.com/Purshow/Purshow_N…)
hellobert retweeted
Found this insanely good vLLM deep dive by @gordic_aleksa 👀
I’ve been learning inference systems for a while, but this one helped connect a lot of the pieces.
It goes from:
→ scheduler + engine core
→ continuous batching + paged attention
→ chunked prefill
→ prefix caching + speculative decoding
→ multi-GPU / multi-node serving
→ latency, throughput + GPU performance
The part i really liked is how it builds the system from first principles instead of treating vLLM like a black box.
If you’re trying to understand what actually happens between generate() and a token showing up on your screen, this is worth reading..
aleksagordic.com/blog/vllm
🥞🍀
hellobert retweeted
Another great resource : github.com/wafer-ai/gpu-perf…
hellobert retweeted
一个请求进入 LLM 推理引擎后，到底经历了什么？
Together AI 工程师 Zain 讲明白了一个请求的一生：一次传播产一个 token，prefill 吃算力、decode 吃显存，KV 缓存、连续批处理层层接力，Agent 循环再把 GPU 榨干。前缀缓存让 100 轮 Agent 计算量从 55 倍降到 10 倍。
hellobert retweeted
You can read it here:
drive.google.com/file/d/12Kl…
hellobert retweeted
Build B200 attention kernel from scratch in CUDA and a little PTX, starting from a naive baseline and reaching 94.4% of FlashAttention-4.
I spent the last couple of months building visual, beginner-friendly progression around 60 diagrams.
We first develop an intuitive understanding of how the naive baseline works, then add one major optimization at a time, pairing each step with detailed diagrams, concise explanations, and code.
For the capstone project, we plug the final kernel into a video-generation model.
iaroslavelistratov.github.io…
hellobert retweeted
Our new AgentX benchmark helped uncover issues that only surface during multi-turn long-context tasks, and provided a real-world benchmark to hillclimb. Read more in our AgentX article (5/5) newsletter.semianalysis.com/…
hellobert retweeted
The real world is multimodal. For AI to understand and recreate it, models need to learn across modalities.
In our latest blog, we show how Miles supports that learning with a shared post-training design for VLMs and diffusion models.
Link in the comments. 🔗
hellobert retweeted
New Blog: Fast Engine Recovery: Sub-Second Engine Restart for SGLang via Weight Cache Daemon.
We built a persistent GPU process that cuts weight loading for Ling-2.6-1T FP8 from ~495s to ~0.63s, by serving post-quantized weights to new engines via CUDA IPC zero-copy.
- ~785× faster weight loading (~495s to ~0.63s on Ling-2.6-1T FP8)
- ~500× on Qwen3-235B FP8
- 93.9% shorter end-to-end boot
- Load + quantize once per GPU, every engine maps the same IPC handles
- Active-standby failover in <1s, no idle GPUs held for replicas
Built together with the Ling Infra team @AntGroup and @Alibaba_Qwen
hellobert retweeted
Excited to team up with @radixark to make large-scale RL data movement faster 🚀
Miles is built for high-performance, large-scale post-training, and Mooncake is now integrated as a rollout data-transfer backend for the fragmented, heterogeneous data moving between rollout and training in disaggregated RL.
On rollout data captured from Miles:
⚡ 10–14× faster remote GET
⚡ 1.2–1.6× faster PUT
By turning fragmented rollout objects into efficient bulk I/O while preserving their original structure, Mooncake helps reduce rollout-to-training handoff latency without changing the RL programming model.
Read more: kvcache.ai/blog/mooncake-rl-…
hellobert retweeted
So privileged to give a talk in the vLLM 小课堂 series about dspark.
Introduced DSpark + how Speculators trains and serves them in vLLM. Amazing questions, and learned more than I shared. The future is open source.
📺 bilibili.com/video/BV18E3u63…
hellobert retweeted
1/ Still looking for a minimalist, high-performance framework for agentic RL research?
Meet Molt — an agentic-first, PyTorch-native reinforcement learning framework with roughly 9K lines of RL code for 700B models.
⭐ github.com/NVIDIA-NeMo/labs-…
hellobert retweeted
Read here: modal.com/blog/unpacking-san…
hellobert retweeted
arxiv.org/abs/2309.04011 is updated
hellobert retweeted
推荐一个老牌小工具：asciinema 🎬
它专门录终端操作，但不是录成视频，而是保存成很轻的文本记录。播放的时候像视频，实际内容还能复制、暂停、快进，放 README、文档、演示材料里都很舒服。
我最近做汇报演示，现场跑命令最怕两件事：网络抽风、手抖打错。后来干脆提前用 asciinema 录好一遍，汇报时直接播放，既像 live demo，又不会翻车。领导看的是效果，我保住的是血压。
这个项目也挺有意思，早期 CLI 是 Python 写的，3.0 直接用 Rust 重写，还加了 terminal live streaming。老工具继续长新牙。
适合这些场景：
录 CLI demo / 写 README 示例 / 做排障复现 / 给新人留操作说明 / 汇报时假装自己很稳 😄
终端世界里的“行车记录仪”，建议常备。
