Google just proved the Transformer is the most expensive version of a much bigger idea. Here's the setup. There are two ways for a model to remember a long context. → RNNs crush everything into one fixed memory. cheap, but they forget. → Transformers keep every token forever. sharp, but the cost grows quadratically. twice the context, four times the compute. For years those felt like two different species. This paper shows they're the same thing at two settings of one dial. The idea is called Memory Caching. Instead of overwriting your memory as you read, you snapshot it. Every so often you save a checkpoint of the memory state and keep it around. When a new token needs to look something up, it can reach back into every saved memory, not just the latest one. So the RNN stops forgetting. Its effective memory now grows with the length of the input. The complexity lands at O(NL). that sits right between the RNN's O(L) and the Transformer's O(L²). N is the dial. Now the part that made me stop. Turn the dial all the way up. Make every single token its own memory checkpoint. Save all of them. You just rebuilt attention. That's what a Transformer is. Memory caching with the segment size set to 1. Every token cached, nothing compressed, maximum cost. The setting where you pay full price. Attention was never the fundamental thing. It's the far end of a spectrum. What that buys you in practice: → Bolt it onto Titans, deep linear attention, or plain linear attention. all of them get better. → Long-context recall climbs hard. on a 16k needle-in-a-haystack test, Titans went from 21 to 32. → It stays subquadratic, so it's far cheaper than a Transformer as context gets long. → You can even bolt it on after training and the model reaches longer contexts than it was trained on. The honest ceiling: a pure Transformer still wins on raw recall accuracy. This closes the gap, it doesn't erase it. But it closes it while spending a fraction of the compute. From the same Google team that built Titans.