Science takes time
Joined September 2018
- Tweets118
- Following459
- Followers200
- Likes1.3K
Pinned Tweet
What algorithm can transformers learn to do in-context learning? Mirror descent 🪞
Our #ICLR2026 paper: 3-layer transformer = one step of mirror descent on the simplex to learn mixture weights.
📄 arxiv.org/abs/2604.10848
📍 Thu, Apr 23 · 3:15–5:45 PM Pavilion 3 · P3-#615
Francesco D'Angelo retweeted
1/ At COLM? Check out our poster
"(How) Learning Rates Regulate Catastrophic Overtraining"
during the second poster session on Tuesday in the Grand Ballroom!
Not at COLM? Read the thread below 👇
Do induction heads just match and copy? Actually, they interpolate n-grams!
📄 Paper: openreview.net/forum?id=BSY7…
📍 ICML 2026 · Hall A · Board #3013
🗓 Tue, Jul 7 · 10:30 AM–12:15 PM KST
👤 @okyksl will be there presenting; come say hi! :)
Francesco D'Angelo retweeted
1/ Did you ever look at some feature represented by a Transformer and wonder: "Why would it even learn that? 🤔 "
We did! Announcing the ICLR'26 paper "Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors"
Francesco D'Angelo retweeted
Swiss AI Visiting PhD Program at EPFL. Deadline Feb 28.
"The program provides a fellowship contribution of CHF 2,500 ($3200) per month, access to the Alps supercomputer, and eligibility for a post-visit continuation grant of up to 50k GPU hours. The call is open to PhD students enrolled outside Switzerland, with applications supported by EPFL PIs contributing to the Swiss AI Initiative. The application closes on February 28."
drive.google.com/file/d/1e0I…
@ICepfl @EPFL_AI_Center
Francesco D'Angelo retweeted
Happy to share that I've started as an assistant professor at @AaltoUniversity and ELLIS Institute Finland!
I'll recruit students via the ELLIS PhD Program ellis.eu/research/phd-postdo… to work on multimodal learning, robustness, visual reasoning... feel free to reach out!
Francesco D'Angelo retweeted
The Maximal Update Parameterization (µP) allows LR transfer from small to large models, saving costly tuning. But why is independent weight decay (IWD) essential for it to work?
We find µP stabilizes early training (like an LR warmup), but IWD takes over in the long term! 🧵
Francesco D'Angelo retweeted
Apparently, our good old weight decay paper made it into the Stanford CS336: Language Modeling from Scratch course...
I'm sometimes nostalgic about my past life when I was doing something other than AI safety :D
Francesco D'Angelo retweeted
🧵What if emergence could be explained by learning a specific circuit: sparse attention?
Our new work explores this bold hypothesis, showing a link between emergence and sparse attention that reveals how data properties influence when emergence occurs during training.
Francesco D'Angelo retweeted
Large language models store vast amounts of knowledge, but how exactly do they learn it?
Excited to share my @GoogleDeepMind internship results, which reveal the fascinating dynamics behind factual knowledge acquisition in LLMs!
arxiv.org/abs/2503.21676
Francesco D'Angelo retweeted
Hao is looking for a PhD position! He is amazing and one of the best students I've worked with — don't miss his application :-)
His master's thesis was published at ICML (arxiv.org/abs/2402.04833) and was nominated for an Outstanding Thesis Award at EPFL. More details below:
Francesco D'Angelo retweeted
Want to understand why deep state-space models / modern recurrent neural networks are so easy to learn?
Come visit @orvieto_antonio and I at our #NeurIPS2024 poster!
📍 Thu 12 Dec 11 am, West Ballroom A-D #6906
📝 arxiv.org/abs/2405.21064
🧵Poster and thread 👇
Francesco D'Angelo retweeted
🚨I'm on the faculty job market this year!🚨
My research focuses on AI safety & generalization. I'm interested in developing technical solutions for ensuring reliability and alignment of advanced AI models, particularly LLM agents that are becoming increasingly capable.
🧵1/4
Francesco D'Angelo retweeted
🚨 So, why do we need weight decay in modern deep learning? 🚨
The camera-ready version of our NeurIPS 2024 paper is now on arXiv (a major update compared to the first version).
Weight decay is traditionally viewed as a regularization method, but its effect in the overtraining regime is quite subtle and its interaction with the implicit regularization effect of SGD plays a crucial role.
In the undertraining regime (e.g., in LLM pretraining), however, the effect of weight decay is totally different: it sets an implicit learning rate schedule for AdamW and enables stable training with bfloat16 precision. This explains why weight decay is still widely used for LLM training with standard optimizers, such as AdamW.
This is joint work with @dngfra, @adityavardhanv, @tml_lab.
Francesco D'Angelo retweeted
❓So why do we need weight decay, really? Don't LLMs directly optimize the population loss and regularization is not needed?
🚨 Our new paper provides a modern look at the old question arxiv.org/abs/2310.04415!
(joint work with @dngfra, @adityavardhanv, @tml_lab)
🧵1/n
Francesco D'Angelo retweeted
🚨Excited to share our new work “Sharpness-Aware Minimization Leads to Low-Rank Features” arxiv.org/abs/2305.16292!
❓We know SAM improves generalization, but can we better understand the structure of features learned by SAM?
(with @dara_bahri, @TheGradient, N. Flammarion)
🧵1/n
Francesco D'Angelo retweeted
🚨 Excited to share our new paper “A modern look at the relationship between sharpness and generalization” arxiv.org/abs/2302.07011!
❓ Do flatter minima really generalize better? Let’s find out!
🧵 1/n
