On this holiest day in the Jewish calendar, I want to share a list of PRs to @sgl_project and @vllm_project focused on low-level performance optimizations for @AIatAMD CDNA3 × Kimi K3 that we at @zroai_ opened over the past year. We touched everything from bugs to new attention kernels. We are now working on CDNA4 (thanks, AMD!). I’d also like to take this opportunity to ask forgiveness from AMD: you ship great hardware, and while your software is not as good, you do your best to improve it, and you are much more welcoming to startups than I gave you credit for. github.com/vllm-project/vllm… github.com/vllm-project/vllm… github.com/vllm-project/vllm… github.com/sgl-project/sglan… github.com/sgl-project/sglan… github.com/sgl-project/sglan… github.com/sgl-project/sglan… github.com/sgl-project/sglan…

Tldr- 1. For the model we tested, Fireworks appears to use off-the-shelf @vllm_project with a relatively simple routing setup 2. As a customer, you can exploit that routing behavior to improve your own inference economics

Introducing HyperQuant: A new Lattice-based compression algorithm that significantly improves LLM key-value cache memory over TurboQuant and OCTOPUS, and with higher accuracy! But it gets better - we apply the same compression technique to weights, outperforming Higgs! Finally, (for the first time ever?), we show how to compress video DiT weights. demo on LTX-2. Easy to integrate: we release a reference kernel code with the paper, under MIT license. This is a calibration-free, post-training quantization. Disclaimer: performance will get much better in next releases. Links and key results in next tweet.

Some time ago, I had the idea to port NVIDIA Physical AI stack to AMD. The motivation was to improve hardware diversity and enable world models and VLAs to run beyond a single ecosystem. We started with NVIDIA Cosmos Predict 2.5-2B. Porting wasn’t trivial: these models are deeply optimized for NVIDIA’s stack. We used this as an opportunity to apply our ROCm kernels. The results were surprising: Both encode and diffusion run faster on AMD Instinct MI300X vs. NVIDIA H200 (FA3) and we still saw significant headroom for further optimization. Quality is unchanged across modalities (validated with WorldJen) To be clear, this is no luck. We have deep experience with diffusion models and AMD GPUs. But this just gives us a good opportunity to get closer to a true hardware-to-hardware comparison, as we work with less software abstractions than usual. Just to give an example, on AMD, memory instructions are async with a hardware queue of ordered pending instructions, enabling concurrent load/store with compute without warp specialization. Bottom line: there are real architectural advantages on AMD, if you take the time to work with the hardware. Note, we did tradeoff ~20% higher memory usage, That being said, AMD has more to give to begin with :) in the coming weeks: AMD versions of Cosmos Transfer and GR00T, an even faster version of Cosmos Predict, and open-sourcing an attention kernel faster than AITER v3 (which is closed-source for some reason? cc: @AnushElangovan )

Introducing BackLite: Attention Backpropagation Acceleration Using Dynamic Sparsity 👀 👊Blog post: moonmath.ai/posts/introducin… 👊Code (open source!) : github.com/moonmath-ai/BackL… 👊Integration example to nanochat: github.com/karpathy/nanochat… It is well known that the attention matrix is highly sparse. Several works have used this sparsity to speed up the forward pass. What if we could also use it to speed up the backward pass? BackLite is a novel algorithm designed to dynamically discover and exploit the sparsity inherent in attention to skip computation while mathematically approximating the gradients through the attention layer. Our idea: Simply track the sparsity in the attention matrix during the forward pass and use it to skip computation during the backward pass. Under the hood: 🌊 Uses the forward pass to track attention matrix tile weights at negligible overhead 🌊 Builds a mask by skipping tiles with cumulative weight less than a threshold 🌊 Skips masked tiles during backward 👉 Same forward, same model, fewer backward FLOPs Drop-in kernel replacement, tested on LLMs and video diffusion models, especially good for long sequence lengths 💪 Disclaimers: Image shows nanochat leaderboard *IF* @karpathy/ @OxyKodit will merge our PR. Yes, there's still much work to do on the code and tests to run. Contributions/questions are welcome.