training llms @ cohere. previously @ tum and inria.
Paris, France
Joined July 2020
- Tweets559
- Following777
- Followers1.4K
- Likes2K
Pinned Tweet
Learning rate schedules seem mysterious?
Turns out that their behaviour can be described with a bound from *convex, nonsmooth* optimization.
Short thread on our latest paper 🚇
arxiv.org/abs/2501.18965
interesting read: What's the matter with France?
paulkrugman.substack.com/p/w…
Personal update: I am excited to join @cohere where I will be working on pretraining from the Paris office.
This also means that my postdoc at Inria comes to an end. I had a great time in the lab, with awesome colleagues, and lots of learning.
Particular thanks to @BachFrancis, @umutsimsekli, @TaylorAdrien for giving me the freedom and space to do research, and for supporting whenever needed.
Fabian Schaipp retweeted
Why would you want to do this?
1) With our law, you get the correct scaling for the optimal batch size with two/three batch sizes per sweep
2) You can also derive scaling laws for *suboptimal* batch size intervals (e.g. which batch sizes waste at most 5% of compute)
very nice paper finding that Muon & SOAP improve training of models for molecular simulations #ICML2026
Poster today at 5pm! #ICML2026
"It's easier to tune the LR for method A than for B."
We tried to formalize this for model-based stochastic optimization methods.
We find a key quantity, called stability index, that describes how stable a (weakly) convex bound is as a function of LR.
📚arxiv.org/abs/2602.09842
Fabian Schaipp retweeted
🎬 Field Notes: is more data always better?
Physical AI is booming, and so are the data providers feeding it. But what actually makes a robot smarter?
Awesome to chat with @fabinsch1, researcher at @inria who just presented his thesis in policy learning for robotics.
🧵
How to scale the batch size in LLM pretraining?
New paper on a scaling law that splits token budget D into training steps and batch size.
💺 arxiv.org/abs/2607.01487
Why would you want to do this?
1) With our law, you get the correct scaling for the optimal batch size with two/three batch sizes per sweep
2) You can also derive scaling laws for *suboptimal* batch size intervals (e.g. which batch sizes waste at most 5% of compute)
awesome blog post! the Apertus project keeps on pushing optimization research for open LLMs.
New research from our MLO Lab @EPFL: Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors.
Magnitude-Direction Decoupling (MD): a simple optimizer tweak, and what we (currently) think is the right way to train efficiently at scale. 🧵
Fabian Schaipp retweeted
Judging optimizer gaps by looking only at language modeling with a fixed batch size is dangerous: one gets only 1/2 of the story. @orientino_ and @ruuustem_10 went beyond. Turns out that for every model, task, and data, there is always a setup where Adam > SGD. 🧵
"It's easier to tune the LR for method A than for B."
We tried to formalize this for model-based stochastic optimization methods.
We find a key quantity, called stability index, that describes how stable a (weakly) convex bound is as a function of LR.
📚arxiv.org/abs/2602.09842
The theory is with SGD as base update step. But all these adaptive step sizes can be used to obtain practical methods. For example, Polyak step sizes in combination with
- Muon (@CrichaelMawshaw): arxiv.org/abs/2510.09827
- ScheduleFree (@aaron_defazio ): arxiv.org/abs/2605.19095