Rust can actually be quite expressive… 🦀

youtu.be/N1rjtDs8blY?is=55Tr… When we think about LLMs today, we usually think of autoregressive, decoder-based Transformers which generate one token, feed it back in, generate the next token, and repeat. That sequential dependency is one of the fundamental limitations of LLM inference today.🧵

Aggregated across the complete benchmark, host + prefetch performed as follows relative to keeping Engram in GPU HBM: • TTFT: +15.57% • TPOT: +10.91% • Output throughput: −14.36% The larger reported KV pool therefore came with slower user-visible service.

Host placement reduced SGLang-reported model-weight residency from 119.043 to 72.201 GB per GPU (-46.842 GB/GPU which is approximately 39.3% reduction). Under the same allocator policy, reported KV capacity rose from 11.84M to 42.70M tokens , a 3.606× increase that could support longer contexts, more concurrent requests, or additional KV-cache headroom, depending on the workload. Host placement increased observed system host-memory use by approximately 195–202 GiB. Server readiness time also rose from 125.04 seconds to between 233.06 and 273.07 seconds.

Some early findings! DeepSeek-V4.1-Flash from @deepseek_ai has 196B Engram parameters that can be moved out of GPU HBM. I tested three placement modes: device, synchronous host, and host with asynchronous prefetch on a node with 4× @NVIDIA B200 GPUs using a pinned @sgl_project build.🧵

I saw the DeepSeek V4.1 Day 0 preview implementation from @sgl_project and was particularly interested in its support for keeping the Engram tables in GPU HBM or placing them in host memory with optional asynchronous prefetch: github.com/sgl-project/sglan… So I’m running a controlled benchmark across these three modes to quantify how much HBM host placement frees, the additional KV-cache capacity that it enables, the resulting host–device transfer latency and throughput costs, and whether asynchronous prefetch can recover the performance lost to offloading. Still running however!

👨‍🍳🚶

DeepSeek has made the KV cache roughly 437× smaller across generations from the deep seek v1. This means you can get close to GPT-6 Astra's performance at a lower cost, first off because you are not paying per token pricing but paying per GPU-hour and because the kv-cache size is greatly reduced you can run up to larger concurrency and longer context sizes!

Here are the benchmark results. OpenDesign has DeepSeek V4.1 Flash at 81.2 basically within touching distance of GPT-6 Astra’s 82.7 and from the technical report it's wayyyy cheaper and faster to run.

Deepseek has cooked again! deepseek-ai/DeepSeek-V4.1-Flash dropped some hours ago with a new Causal Encoder–Decoder architecture, native multimodality and very aggressive KV-cache compression. This might actually be a new sign of a genuinely new frontier architecture direction since the decoder-only models by GPT-2 took over. Crazy times! 🧵