@tmpethick

Uncertainty in the face of optimism

pethick.dk
Joined July 2011
  • Tweets194
  • Following91
  • Followers334
52Media
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking: What Schatten-p norm to use? Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
3
17
6
117
23,287
We now have a poster! xitter.cf/tmpethick/status/20668…
Thomas Pethick@tmpethickJun 16
1/ Let me chip in on the recent “which optimizer rules them all” discussion with a somewhat more moderate take, asking: What Schatten-p norm to use? Turns out the answer is regime dependent! Specifically, even when smooth in Schatten-∞, Muon is not necessarily the best choice.
1
1
19
2,297
The slides from my talk at @Inria this week on "when to use what Schatten-p norm in deep learning" are now available here: pethick.dk/talks/2026_Inria_…
4
60
4,151
Thomas Pethick retweeted
Thomas Pethick@tmpethickJun 18
Replying to @TedNoNumbers @Creative_Math_
Yes, I think there is something interesting to be said here: take a look at @leloykun's blogpost leloykun.github.io/ponder/mu… "So, in a sense, you can interpret Muon with the old coefficients as doing steepest descent under the Schatten-32 norm"

Squeezing 1-2% Efficiency Gains Out of Muon by Optimizing the Newton-Schulz Coefficients

Simply switching to Muon can already get you 2x efficiency gains. But you can squeeze out an extra 1-2% by optimizing the Newton-Schulz coefficients.

leloykun.github.io
1
2
4
253
Replying to @tmpethick @konstmish @kellerjordan0 @jxbz @leloykun @_arohan_ @gowerrobert @vyasnikhil96 @SebastienBubeck @JohnCLangford @ed_gorbunov @tonysilveti @DimitrisPapail @damekdavis @bremen79
I might also add that it should be possible to draw similar conclusions for Shampoo and PSGD parameterizations for certain ranges by using @varunneal's observation: xitter.cf/varunneal/status/20665…
varun
@varunnealJun 15
Replying to @varunneal @evaninwords
in fact, whitening mode psgd w criteria c(P; g, v) = P[g,g] + P^{-1}[v,v] can be replaced w "generalized" form c_k(P; g, v) = P[g,g] + P^{-k}[v,v] which has fixed pt corresponding to spectral power (k-2)/(k+2) SoftMuon p = shampoo (p-1)/4 = Whitening PSGD 2(1+p)/(1-p)
1
1
8
838
The theory also seems to be consistent with this observation on Nesterov momentum from @_arohan_ xitter.cf/_arohan_/status/206474…
rohan anil
@_arohan_Jun 10
Replying to @evaninwords
“Shampoo falls halfway between Muon & Adam” was the very confident claim. This has been addressed. It had to be done. Nesterov largely have not been useful in language models from my experience and now for that I got the answer as well. Not enough effort has gone to these submissions. This is why we do tuning budgets anf made as fair comparison as possible @zacharynado arxiv.org/abs/2502.15015

Accelerating Neural Network Training: An Analysis of the AlgoPerf...

The goal of the AlgoPerf: Training Algorithms competition is to evaluate practical speed-ups in neural network training achieved solely by improving the underlying training algorithms. In the...

arxiv.org
1
351
Thomas Pethick retweeted
Keller Jordan
@kellerjordan0Jun 12
Result #32: @mihai673 has achieved a 30-step improvement over the old 2026/05/09 record by adding a SODA (Pethick et al. 2026)-style anchor towards init. It is unknown whether this technique can also improve the current record. 2/5
1
3
18
1,720
Thomas Pethick retweeted
catid
@MrCatidMay 27
SODA-AMUSE+Gram+PMuon is the recipe that wins consistently on 4/4 codex GPU instances. PMuon is a new idea: github.com/zzp1012/modded-na… AMUSE is Muon + ScheduleFree: arxiv.org/abs/2605.22432v1 SODA is a wrapper that's more popular: arxiv.org/abs/2605.11172 Gram is a lesser known Gram-Newton-Schultz optimization: github.com/Dao-AILab/gram-ne…
1
4
12
2,055
Load more