We discovered a new geometric approach to self-attention! 1) We represents weights as functions in GPU memory, rather than materializing weights. This allows the first model that can sample new layers at inference from a curve(8 for a phone, 24 for a server). The function is constructed from basis matrices that allows us to sample layers on the curve dynamically, even based on the curvature where there's most information. 2) Hierarchical state beats transformers(NanoGPT) KV cache with 14,000x less state and 50% better CE at 2K context. 3) We use forward pass perturbations between pre-training steps to optimize the model further without tokens. 4) We cache experts using a “bundling” trick based on use-case to further compress what we need to run on GPU. The goal is to accelerate everyone, including all the auto-researchers, and run frontier models on laptops. I've been working with a Google researcher and my former professor on this the past few months. The week Fable dropped, our progress exploded with auto-research, but we were still working on publishing a paper. With the clear explosion of decentralized auto-research on Github, I’m open sourcing our work soon :) It’s very similar to Integer Mult bounds, as we explore new geometries to compress data, so if you’re interested in helping let me know! I'm testing auto-research.md rails now and plan to publish the Github by the end of the week!