There is a beautiful dual view of gradient descent on dense linear weight layers: A linear layer trained by GD computes exactly the same function as its initial weights plus linear attention over its own training history. Each test input attends to every input the layer received during training ("keys") and recombines the backpropagated errors stored assoc. with them ("values"). This intriguing connection has been discovered and revisited before but is unfortunately still not really well-known! If you want a new perspective on one of the most fundamental things we do when training modern neural networks, I highly recommend studying this dual view, and thoroughly thinking about its implications. Taking this perspective seriously, we replaced the linear attention with more powerful kernels, such - obviously ;) - softmax attention. This (not trivial at all!) led to a new paradigm for neural network learning that we call Retrieval-Centric Deep Learning (RCDL): neural networks whose basic building blocks aren't classic linear weight matrices but growing, nonparametric databases of inputs and backpropagated errors, read out by nonlinear attention variants. Via the RCDL approach, we were furthermore able to find cool, insightful derivations for known ml tools such as the FOOF optimizer! In our new paper, we describe this RCDL in detail and study its behavior on simple learning tasks - it performs surprisingly well!  And of course, we also address the obvious current limitation: Linear growth in memory as the number of training points increases. To circumvent this, we propose some early ideas using a hybrid parametric/nonparametric approach. We are particularly excited to see what tricks can be found to adapt these growing layers to today's hardware constraints. But after all - this work was originally motivated by a simple idea: Compute is getting cheaper and cheaper, hence are there methods that seem wildly expensive today, but may be affordable and extremely powerful in the future? We think this RCDL paradigm offers a promising path :) There is a ton of possible follow-up work that can be done now and we are very excited to see what people will try and do with this new retrieval centric learning approach!

Scale ML is resuming this Fall with Alok from Jane Street! We will go over how to do data mixture experiments at scale! 🔥 Sign up for the zoom link via the website.