notes on A Defense of the Quadratic Model link - arxiv.org/abs/2607.21716 ------------------------------------------------ it may be possible to use a simple model to predict how a model improves in late stages of training. perhaps the most interesting paper for LLMs in a while to me. how good is that claim? let us see. when we train a language model we are trying to change millions of knobs with each feedback to minimise our errors in prediction. but the easiest way to visualise this for this paper discussion is that you imagine a traveller who has to traverse a really unpredictable terrain all the way to the lowest point possible. now this paper asks that if we were to replace this uneven landscape with a simple bowl then how successful would we be in creating a copy of the traveller that can improve similar to the real uneven landscape? in the early and middle stages of training, this simplistic mapping fails quickly however near the end when the learning rate is slowly getting reduced it can match how the model would improve for several percentages of the training budget (5-10%). this does not work though when the learning rate is constant and works in the case of cosine decay and the model error prediction mapping fails even in the late stages of training in the constant learning rate case. this may be explainable with the help of mainly two changes. first, the learning-rate schedule makes the model take much smaller steps, so it remains inside the tiny neighbourhood where a simplistic local map should work because we expect no major changes in the terrain. second, learning may be saturating iin the late stages of training and the way it would present itself is where most common patterns are already learned. this means lesser and lesser drastic changes to the model landscape. the remaining weight changes could consequently concentrate into a low-rank, slowly rotating set of directions here. and the complete landscape can remain complicated while the narrow corridor in late stages of training actually used by the optimizer becomes simple and stable enough to resemble a bowl. the landscape itself not getting much drastic updates and the traveller slowing down allowing stable prediction for a budget of tokens where the landscape walk is actually much smaller are my intuitive guesses here but i think we can perhaps get creative with the predictable narrow training budgets part.