PhD student in machine learning @CambridgeMLG.
Joined October 2021
- Tweets342
- Following56
- Followers739
- Likes869
Pinned Tweet
1/14 Is Muon “better” than Shampoo?
We argue that their relationship parallels Adam's relationship with Signum. Analogous to @lukas_balles and Hennig’s (2018) decomposition of Adam into element-wise scaled Signum, we can decompose Shampoo as left- and right-adapted Muon.
Runa Eschenhagen retweeted
We've just released laplace-torch v0.3. Highlights: Function-space posterior approximations through ELLA and VaLLA! Awesome work by @LAOrtega_!
github.com/aleximmer/Laplace
Runa Eschenhagen retweeted
An awesome opportunity to work on important questions in scaling with someone who thinks deeply about them. Spencer was a great host, and I really enjoyed my time at GDM with the SF team!
I'm hiring a Student Researcher to work on scaling laws at Google DeepMind! Project is for 19 weeks, starting winter/spring '27 (ideally: January '27), in-person in downtown SF. If you're interested, fill out this form: forms.gle/9MKo9ciBmp5wsyiU7
Runa Eschenhagen retweeted
I recently moved to TU Darmstadt (Germany) and am currently hiring PhD student through ELISS.
ellis.eu/news/ellis-phd-prog…
If you interested, please apply and choose me as a potential advisor. Topics include Continual/Distributed/Federated/Active/Transparent AI.
Runa Eschenhagen retweeted
Rich Turner is hiring a Research Associate at Cambridge to work on AI for weather forecasting 🌥️
Strong deep learning background required, but no prior weather experience is fine. Especially relevant: spatiotemporal modelling, PDEs, time series
Runa Eschenhagen retweeted
📣 I'm accepting applications for funded grad study in my group (Fall 2027). Details and instructions here: agustinus.kristia.de/opening…. Apply by end of September!
Runa Eschenhagen retweeted
1/8 How well does the simplest model in optimization theory, the quadratic, actually describe LLM pretraining? Surprisingly well. We linearize a 150M model at checkpoints throughout training and the local Taylor expansion tracks the true loss for up to 10% of the training budget.
Runa Eschenhagen retweeted
hello there the jacobian conjecture is false thanx to my close friend akhil for asking about it and my other close friend fable for working during the world cup final
((1+xy)^3 z + y^2 (1+xy) (4+3xy), y + 3 x (1+xy)^2 z + 3 x y^2 (4+3xy), 2 x - 3 x^2 y - x^3 z): \C^3\to \C^3, has jacobian determinant -2, and sends (0, 0, -1/4), (1, -3/2, 13/2), and (-1, 3/2, 13/2) to (-1/4, 0, 0)
Runa Eschenhagen retweeted
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking:
What Schatten-p norm to use?
Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
Runa Eschenhagen retweeted
Something I see getting ignore over and over: if you take Marek et al. (2025) seriously, a ‘vanilla’ continuous-time story is not enough to explain Muon's advantage over Adam in LLM training.
You have to account for discretisation error to explain the improved convergence.
Once both sides define what they mean by Shampoo and Muon before debating their relationship, there will be nothing left to debate.
One motivation for me to add these optimizers was seeing @_arohan_, a senior researcher whose contributions I respect, repeatedly claim that Muon is Shampoo.
IIUC, his argument is that if we disable accumulation in both Muon and Shampoo, then they become...
4/6
This is a very common problem in philosophical debates, but in this case things are actually easy to define precisely. Reminds me of this:
Runa Eschenhagen retweeted
The recent Microsoft AI report noted that too much learning rate decay during pretraining hurts post-RL performance. This is actually just the latest of several papers this year pointing out that small learning rates can be harmful in LLM pretraining. (Thread)
Runa Eschenhagen retweeted
Awesome to see this work from Marin.
One detail I’m especially pleased about: their final recipe adapts ideas from our Complete(d)P work, especially horizon and batch-size hyperparameter transfer, and they verify it scales to very large scales.
Not only do we want to train a good model, we want to know it'll be good before we even start training.
About a month ago, the Marin team launched a 129B (16B active) 1e23 FLOPs MoE run and preregistered a loss of 2.252. The run finished this past week and landed at 2.234.
xitter.cf/percyliang/status/2044…
Runa Eschenhagen retweeted
Guys, you need London training data. Trust me, the model won’t generalise. We clean on the left. Pls
Today, we're launching shift. We're starting by cleaning your apartment in New York City, for free.
Here's how it works. Book a shift cleaning. A vetted shift operator comes to your home wearing one of our devices. They clean. They leave. You pay nothing.
In exchange, we record the cleaning. Robotics is being built on data about how people do daily tasks, and the value of that recording is what funds the service. Anything personal in it is anonymized before the recording is processed.
By now, you have heard about the shift to AI more times than you can count. About the shift toward you, the part where you actually feel it, you have heard almost nothing. Shift is what starts to make it concrete, in specific cities, with specific services.
Today, cleaning in New York. Soon, handymen, repairs, and errands across the globe. And this is just one side of shift, with more on the way.
Comment “shift” and we’ll send you an early access link.
Runa Eschenhagen retweeted
🚨 New Paper 🚨
ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models
A few modifications to Schedule-Free Learning make it completely LR tuning free, and allow it to greatly outperform schedules for long duration training!
arxiv.org/abs/2605.19095v1
Runa Eschenhagen retweeted
Ownership is the biggest factor in quality research: do the drivers of the work actually feel a stake in its success, a sense of responsibility and autonomy, resilience, and a desire to make a contribution that transcends a conference publication or an immediate reward signal?
Runa Eschenhagen retweeted
Many believe that optimizers like Muon perform well because of their connection to spectral geometry.
But this is not the case!
In fact, replacing the spectrum of the update with random or even inverted singular values performs remarkably similar!
arxiv.org/abs/2605.11181
