6 KV cache quantization approaches, explained visually: Quantizing model weights doesn't automatically shrink the KV cache. The cache stores activations generated during inference, and its memory grows with retained tokens and concurrent requests. KV quantization reduces the bytes used to store those activations. For instance, moving from 16-bit to 8-bit halves the raw KV payload, before scales, metadata, and any high-precision buffers. The diagram covers six approaches. In practice, several approaches can be combined within the same implementation. 1. FP8 cache The KV cache normally stores each number using 16 bits. FP8 reduces this to 8 bits, so the numbers take up half the space. A scaling factor adjusts their range before storage. The serving engine needs an attention kernel that supports this format. 2. INT8 / INT4 cache These methods convert cached numbers into 8-bit or 4-bit integers. INT4 uses less memory, but has fewer values to choose from, so rounding introduces more error. During attention, the kernel converts the stored integers back into approximate versions of the original numbers. 3. Quantization granularity Quantization uses a scale to convert numbers into a smaller format. One scale can serve the whole tensor, or separate scales can serve individual channels, tokens, or small groups. Smaller groups fit their numbers more closely. But each group needs its own scale, which also takes memory. 4. KIVI-style quantization Keys and values don't have the same pattern of large numbers, so KIVI compresses them differently. For keys, it groups numbers from the same channel across tokens. For values, it groups numbers within each token. This helps preserve accuracy even at 2-bit precision, without fine-tuning the model. 5. Mixed precision This approach keeps recent tokens' keys and values at higher precision and compresses older entries. Attention reads both parts. Keeping some entries uncompressed avoids rounding errors for those entries, but uses more memory than compressing the entire cache. 6. Non-uniform / outlier-aware quantization Uniform quantization rounds numbers to evenly spaced levels. Non-uniform quantization changes the spacing to better represent the numbers that matter. Outlier-aware methods handle unusually large numbers separately. KVQuant stores these outliers in a separate structure, so they don't force the remaining numbers into a range that's too wide. A smaller cache can fit longer contexts or more requests on the same GPU. It doesn't guarantee faster decoding, because compressing and reading the numbers also takes work. Compare each method against an unquantized cache to check how much memory it saves, whether decoding gets faster or slower, and whether answer quality drops. If you want to dive deeper, I wrote a KV Cache Engineering article covering quantization and other ways to reduce cache memory, with code examples and their trade-offs. Read it below.