PagedAttention, clearly explained. (how vLLM manages KV cache like an operating system) every request a serving engine handles needs GPU memory for its KV cache, the key and value vectors stored per token so later tokens can attend over the prompt without recomputing it. the engine has no way to know how many tokens the request will generate. the simple approach is to reserve one contiguous slab per request based sized for the maximum allowed length (max_token_size). for example, a request with max_token_size = 4,096 tokens gets 4,096 slots the moment it arrives and holds them until it finishes, even if it stops at 40. the left side of the graphic below shows what that does to memory. → a large part of every slab is reserved but unused. those slots belong to one request until it finishes, and most of them never get written. → the grey pieces between slabs are fragmentation gaps. as requests of different sizes come and go, the free memory breaks into fragments that are each too small for a new slab, even when the total free space would fit several requests. a new request needs a contiguous slab, none of the gaps fit one, and the reserved slots cannot be taken back, so it waits while the GPU holds mostly empty space. the vLLM team measured 60 to 80% of KV cache memory wasted this way in existing serving systems. they introduced PagedAttention which is inspired from how operating system uses RAM. instead of one slab per request, the KV cache is handed out in blocks. a block is a fixed-size chunk of GPU memory that holds the keys and values for a fixed number of consecutive tokens, in vLLM the default value is 16. inside it sits the full KV state for those 16 tokens across every layer and every KV head. at startup the engine measures the GPU memory left after loading the weights and carves all of it into a pool of numbered physical blocks. that pool is the grid in the graphic below, and it is the only KV memory that will ever exist. the bottom row shows how a request uses that pool. → a request arrives and its sequence is split into logical blocks, L1 for tokens 1 to 16, L2 for 17 to 32, and so on. a logical block is only an index into the sequence, it holds nothing itself. → each logical block is assigned a physical block from the pool and the pairing is recorded in the request's block table. the assignment happens only when the previous block fills, so no request holds memory it has not written yet. → at each decode step the attention kernel reads the block table and gathers keys and values from whichever physical blocks they landed in. a request's tokens no longer need to be next to each other in memory, which is why consecutive logical blocks can sit at scattered grid positions. every physical block is the same size, so any free one serves any request and fragmentation gaps stop forming. when a request finishes, its block numbers go straight back to the pool for the next request to take. waste can never be more than the unfilled tail of a single 16-token block, which is how it drops under 4%. the reclaimed memory becomes batch size, meaning more requests generating at the same time on one GPU. that is where the 2 to 4x throughput gain comes from. each generation step is limited by how fast the model weights can be read from memory, and a bigger batch spreads that read across more tokens. the model itself is untouched. whatever attention variant it was trained with runs the same way under PagedAttention, because the change lives entirely in how the serving engine hands out memory. i wrote the full breakdown of every attention mechanism, from multi-head attention through FlashAttention and sparse attention, up to PagedAttention and RadixAttention. the article is quoted below.

The easiest way to find out which models you can run on your computer: Just run: - 𝗻𝗽𝗺 𝗶 -𝗴 @𝗺𝗮𝗴𝗻𝗶𝘁𝘂𝗱𝗲𝗱𝗲𝘃/𝗰𝗹𝗶 - 𝗺𝗮𝗴𝗻𝗶𝘁𝘂𝗱𝗲 𝘀𝗲𝘁𝘂𝗽 It profiles your machine and ranks the models across: - Speed - Accuracy - Intelligence - Memory required Finally, you can choose your favorite harness (Pi, OpenCode, Claude Code, Codex, etc.) to run with it. Get started here: github.com/magnitudedev/magn… (don't forget to star 🌟) I also wrote a detailed article on running your favorite harnesses with local models. The article is quoted below.