Researcher at INRIA (CELESTE team). Former Postdoc at EPFL and PhD student at ENS Paris-Saclay. Focusing on theory of neural networks and multiplayer bandits
Paris, France
Joined November 2017
- Tweets65
- Following279
- Followers226
- Likes285
Another thread on the theory of fine-tuning neural networks!
Very proud of this recent work with Alexandre Declèves and Nicolas Flammarion on fine-tuning diagonal linear networks (DLNs) from pre-trained weights.
With label noise, the final interpolating estimator overfits, so early stopping becomes necessary.
Building on the saddle-to-saddle dynamics of Pesme and Flammarion, we provide statistical guarantees for a suitably tuned early-stopping criterion.
Overall, this work shows how gradient-based fine-tuning can efficiently reuse information inherited from pre-training, with strong sample complexity guarantees for the downstream task.
Hope you’ll enjoy reading it!
arXiv: arxiv.org/abs/2609.34756
I haven’t posted here in a while, but this feels like a good occasion to come back and share some recent work I’m very proud of!
Two closely related works with Nicolas Flammarion on the theory of fine-tuning neural networks 🧵
We show that this two-timescale algorithm recovers the new feature while preserving the pre-trained ones in its representation.
Even better, it only requires order d fine-tuning samples to do so, independently of the number m of pre-trained features.
I hope you’ll enjoy reading it as much as we enjoyed working on it!
arXiv: arxiv.org/abs/2609.34667
Etienne Boursier retweeted
We are thrilled to say that NeurIPS@Paris is back for a 4th edition on the 4th and 5th of December 2024 at @Sorbonne_Univ_. A great occasion to meet + discuss recent advances in ML in central Paris!
More info: neuripsinparis.github.io/neu…
Registration: forms.gle/Fc4nbeW6ubahYbJn6
Training dynamics of ReLU networks is back! Many works point out to a mysterious early alignment phase. While this phase has obvious perks for implicit bias, it can also lead to harder optimization and even convergence towards spurious stationary points. Let me explain 🧵
In summary, small init lead to good implicit bias, but potential optimization failures. On the other hand, large init (NTK) are known to yield easy optimization, but bad implicit bias. Of course, intermediate init might get the best of both worlds, but are harder to analyze.
As always, big thanks to my coauthor Nicolas.
More in the paper: arxiv.org/abs/2401.10791
Code and animations: github.com/eboursier/early_a…