Day 5: Inference Engineering Today was more about ML than inference tbh. Got caught up reading the Gated Delta Net and Kimi Delta Net attention papers. I think that this is the future of models (no chance OAI and Anthropic don't adopt these linear memory architectures) so its really important to understand from first principles. GDN and KDA (used by Qwen and Kimi respectively) use 3 linear layers followed by one MLA layer; so you technically still have a KV cache but it is compressed (bc of MLA) and of that only 1/4th its typical size (bc of it only being every fourth layer). It is hard to overstate how large of a difference this causes for inference. So much of model serving is memory/KV management and if KV computation is no longer quadratic in size you are able to serve across far higher batches bc you are less restricted by the memory wall. In the linear layers, you have a single dim(k) x dim(v) matrix that, upon each step, is degraded (GDA degrades the entire matrix by a scalar, KDA across feature dimensions by a learned vector) and then add on a vk^T outer product. It is genuinely absurd to me how few people talk about how large of a deal these model arch updates are; especially for them to be open source. Such a big part of being an engineer is constantly refreshing your knowledge and keeping up with the cutting edge (although I admittedly first read about GDA in March, I wrote it off naively). Also read a lot about Looped Transformers. Rumor has it a lot of future open and closed source models are gonna be based on this arch. Basic idea is that, instead of doing 1 full pass through 32 layers, each with independent Attention and FFN weights, you do 4 'looped' passes through 8 layers of Attention and FFN weights. So you use the same amount of compute, but save 4x (this parameter is adjustable) the weight memory. Pair this with linear attention and you are using soooo much less memory than a vanilla transformer (Say GQA with full KV cache) would use and this is still even *before* quantization. The fact that the model passes through the same set of weights a certain number of times means each layer receives an updated residual vector on each loop, meaning the residual has layers to be corrected/shaped into a more accurate area of the embedding space. These continual pass throughs are very analogous to training epochs (imagine doing a single epoch in training and thinking you were chillin). This also allows each recurrent linear memory state to receive a new residual input and thus get new k and v projections from their respective W_K and W_V matrices. These are both such deep papers and this is only the surface so I hope you understand why working my way through these two papers took me more or less all day arxiv.org/abs/2605.22791 arxiv.org/abs/2510.26692