@tmpethick

Uncertainty in the face of optimism

pethick.dk
Joined July 2011
  • Tweets194
  • Following91
  • Followers334
52Media
Include

Only show posts containing:

Exclude

Hide posts containing:

Time range
-
Minimum likes
Replying to @xidulu @laurence_ai @ShaneBergsma
This looks exactly like the O(1/steps) schedule we theoretically arrive at with SODA arxiv.org/pdf/2605.11172 I should check out your papers!
1
2
8
478
Replying to @kalomaze
It might be interesting for you to look at arxiv.org/pdf/2605.11172 where we weight decay towards the initialization - we'll soon put out the updated accepted version which applies this idea to fine-tuning (where init=pre-trained model)
1
4
114
Replying to @tmpethick @ruuustem_10 @tonysilveti
Ah I see, it’s the muKL regardless of normalization that does it, e.g., the O(k^-1/3) achieved here for SGD proceedings.neurips.cc/paper…
1
44
Replying to @ruuustem_10 @tonysilveti
You mean in their experiment and without normalization/clipping? AdamW would be ~consistent with your theory, but SGDM does not fit into FW so how do we explain?
1
66
Replying to @tonysilveti
Interesting! I'd be curious to see if they use normalization for SGD. If yes it would be consistent with your theory (since it would be Frank-Wolfe). Otherwise I would expect the exponent to potentially be different as in eg xitter.cf/tmpethick/status/20668… where "SGD" is not normalized
Thomas Pethick@tmpethickJun 16
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking: What Schatten-p norm to use? Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
1
2
333
Replying to @stochasticchasm
It might be interesting for you to check out angular muown and magnitude-direction decoupling that came out concurrently arxiv.org/pdf/2606.23637 arxiv.org/pdf/2606.25971
1
1
19
Replying to @SmartPig_Joe
There’s a simple argument for why aggressive wd/normalization hurts for the last layer (if one does not compensate by introducing a learnable scalar) xitter.cf/tmpethick/status/20545…
Thomas Pethick@tmpethickMay 13
It is common to exclude the unembedding layer from using weight decay, but why? Essentially, applying weight decay to the last layer will constrain the logits of the model and consequently prevent high-confident output. 👇 xitter.cf/tmpethick/status/20541…
2
181
We now have a poster! xitter.cf/tmpethick/status/20668…
Thomas Pethick@tmpethickJun 16
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking: What Schatten-p norm to use? Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
1
1
19
2,297
Replying to @borisdayma
How come you chose to use fixed lr and not e.g. linear decay?
1
54
The slides from my talk at @Inria this week on "when to use what Schatten-p norm in deep learning" are now available here: pethick.dk/talks/2026_Inria_…
4
60
4,151
Replying to @DengShenyang24 @wen_kaiyue
A theoretical take that might be interesting to you: xitter.cf/tmpethick/status/20668…
Thomas Pethick@tmpethickJun 16
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking: What Schatten-p norm to use? Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
2
224
Thomas Pethick retweeted
Thomas Pethick@tmpethickJun 18
Replying to @TedNoNumbers @Creative_Math_
Yes, I think there is something interesting to be said here: take a look at @leloykun's blogpost leloykun.github.io/ponder/mu… "So, in a sense, you can interpret Muon with the old coefficients as doing steepest descent under the Schatten-32 norm"

Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients

Simply switching to Muon can already get you 2x efficiency gains. But you can squeeze out an extra 1-2% by optimizing the Newton-Schulz coefficients.

leloykun.github.io
1
2
4
253
Replying to @JohnCLangford @konstmish @kellerjordan0 @jxbz @leloykun @_arohan_ @gowerrobert @vyasnikhil96 @SebastienBubeck @ed_gorbunov @tonysilveti @DimitrisPapail @damekdavis @bremen79
Thanks for trying this! The update looks somewhat different from the proposal. I would maybe start with QDWH or power iterations as eg used in Dion originally and not worry about per iteration cost to begin with
1
2
152
Replying to @tmpethick @kellerjordan0 @JohnCLangford @konstmish @jxbz @leloykun @_arohan_ @gowerrobert @vyasnikhil96 @SebastienBubeck @ed_gorbunov @tonysilveti @DimitrisPapail @damekdavis @bremen79
Btw I should say that I find it quite remarkable/impressive that you put together a method that is so strongly backed by theory in retrospect (and thus very amenable to tweaking)
1
234
Replying to @kellerjordan0 @JohnCLangford @konstmish @jxbz @leloykun @_arohan_ @gowerrobert @vyasnikhil96 @SebastienBubeck @ed_gorbunov @tonysilveti @DimitrisPapail @damekdavis @bremen79
Thanks for sharing! The claim I was trying to convey support of is somehow more narrow: within class of Schatten-p norms, smaller p can be optimal. There’s still interesting work left in generalizing to/understanding Shampoo/SOAP, PSGD etc
1
1
199
Replying to @tmpethick @TedNoNumbers @Creative_Math_ @leloykun
Thanks to @tonysilveti for making me aware of this post originally
2
77
Replying to @TedNoNumbers @Creative_Math_
Yes, I think there is something interesting to be said here: take a look at @leloykun's blogpost leloykun.github.io/ponder/mu… "So, in a sense, you can interpret Muon with the old coefficients as doing steepest descent under the Schatten-32 norm"

Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients

Simply switching to Muon can already get you 2x efficiency gains. But you can squeeze out an extra 1-2% by optimizing the Newton-Schulz coefficients.

leloykun.github.io
1
2
4
253
Replying to @orvieto_antonio
Thanks for sharing! I think the result complements your work quite well because it is essentially interpolating between *unnormalized* “SGD” to “signSGD” (normalized) - I comment on this somewhere in the method section
1
28
Replying to @tmpethick @konstmish @kellerjordan0 @jxbz @leloykun @_arohan_ @gowerrobert @vyasnikhil96 @SebastienBubeck @JohnCLangford @ed_gorbunov @tonysilveti @DimitrisPapail @damekdavis @bremen79 @varunneal
The theory also seems to be consistent with this observation on Nesterov momentum from @_arohan_ xitter.cf/_arohan_/status/206474…
rohan anil
@_arohan_Jun 10
Replying to @evaninwords
“Shampoo falls halfway between Muon & Adam” was the very confident claim. This has been addressed. It had to be done. Nesterov largely have not been useful in language models from my experience and now for that I got the answer as well. Not enough effort has gone to these submissions. This is why we do tuning budgets anf made as fair comparison as possible @zacharynado arxiv.org/abs/2502.15015

Accelerating Neural Network Training: An Analysis of the AlgoPerf...

The goal of the AlgoPerf: Training Algorithms competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. In the...

arxiv.org
1
351
Replying to @brunorganised @thomasahle
You might want to take a look at this work - you can extract the large batchsize requirement simply by optimizing the oracle complexity: xitter.cf/tmpethick/status/20668…
Thomas Pethick@tmpethickJun 16
Replying to @tmpethick
5/ In the process we derive a batch size scaling rule for arbitrary Schatten-p norms, which interestingly: - for p=∞ recovers the BST scaling rule from xitter.cf/ruuustem_10/status/203… - and suggests small batch size for Euclidean ala xitter.cf/micahgoldblum/status/1… and xitter.cf/orvieto_antonio/status…
1
74
Load more