I implemented Google's TurboQuant paper (ICLR 2026) in llama.cpp with Metal kernels for Apple Silicon. 4.9× KV cache compression. Working end-to-end on M5 Max with Qwen 3.5 35B MoE and Qwopus v2 27B. Speed needs work (unoptimized shader), compression target met. Repo: github.com/TheTom/turboquant… **Note**: as you'll see from the git when I saw "I" it's in conjunction with claudecode and codex. Just lots of steering and babysitting.

🧵As 2026 unfolds, sparse MoE models are emerging as the new backbone for high-throughput inference and Agent workloads. Step 3.5 Flash from StepFun @StepFun_ai stands out with its efficient attention mixture design. Here’s a technical deep dive from Zhihu contributor kaiyuan 👇 🤖 Model Overview • Open-source MoE LLM built for high-throughput inference & Agent scenarios • Matches or exceeds leading models in reasoning, coding, and Agent benchmarks • Delivers speed-quality balance via sparse MoE routing + Multi-Token Prediction (MTP) 📐 Core Architecture • Backbone: Transformer + MoE | Total params: ~196B | Active params: 11B/token • Attention mix: GQA + Sliding Window Attention (SWA) + Full Attention • Routing: 288 experts | Top-8 activation per token • Context window: 256K (262144 max sequence length) ⚙️ Key Configs (config.json) • Layers: 45 | Hidden dim: 4096 | Attention heads: 64 • Sliding window: 512 | Max sequence length: 256K ✨ Standout Features • Sparse MoE Routing: 288 experts with Top-8 selection → cuts compute without losing capacity • Dual Attention Mechanism: SWA for efficient local modeling (512-window) + Full Attention for global context • MTP Acceleration: Faster decoding for real-world interactive throughput ✅ Final Takeaway Step 3.5 Flash proves that sparse MoE architectures can deliver enterprise-grade performance for long-context, high-throughput applications without proportional resource growth. #AI #Engineering #Tech #LLM #Agent #StepFun 🔗 Full article(CN):zhuanlan.zhihu.com/p/2021161…

Introducing TurboQuant: Our new compression algorithm that reduces LLM key-value cache memory by at least 6x and delivers up to 8x speedup, all with zero accuracy loss, redefining AI efficiency. Read the blog to learn how it achieves these results: goo.gle/4bsq2qI