@LinYorker

Postdoctoral fellow at @VectorInst. ML PhD at UBC. Mathematical and computational structures for ML. Geometric and algebraic methods.

 Orlando Florida
yorkerlin.github.io/
Joined August 2020
  • Tweets115
  • Following40
  • Followers327
30Media
#ICML2024 Can We Remove the Square-Root in Adaptive Methods? arxiv.org/abs/2402.03496 Root-free (RF) methods are better on CNNs and competitive on Transformers compared to root-based methods (AdamW) Removing the root makes matrix methods faster: Root-free Shampoo in BFloat16 /1
4
16
2
63
13,002
I still remember the comparison between SGD and Adam a few years ago. Empirical results may not support that. So, wait and see just like the SGD vs Adam lesson.
jianlin.su@Jianlin_SSep 28
Slow is Fast: Is Muon the end of Shampoo? kexue.fm/archives/11917 Orthogonal Slowest Descent — from a convex optimization convergence bound — washes away any Kronecker preconditioner's eigenvectors, leaving only eigenvalues to balance the gradient's spectrum — leads to Muon.
5
221
On one hand, it is essential to tune baseline methods well on a model. On the other hand, it may be better to avoid using a model/architecture that has been modified and optimized for a single method for 1.5 years.
Konstantin Mishchenko
@konstmishJun 11
I just submitted a PR to modded-nanogpt with better hyperparams. With them, Muon can reach the target loss after 3250 steps instead of 3325. Always tune your baseline well when doing research. Weak baselines can make any idea look promising
1
1
7
1,237
Some initial steps to make Shampoo and SOAP faster arxiv.org/abs/2605.26327 We are working on further improvements.

Reparametrizing Shampoo and SOAP for Subspace Basis Updates and...

Shampoo-based methods, such as KL-Shampoo and SOAP, have demonstrated strong performance in training neural networks and rely on QR decomposition. Because existing QR implementations require...

arxiv.org
1
1
70
We will make Shampoo/SOAP, including KL-Shampoo/KL-SOAP, faster. Our goal is to match Muon's runtime while maintaining Shampoo/SOAP's strong per-step performance. Stay tuned for new updates.
Lucas Nestler
@ClashlukeMay 2
KL Shampoo and KL SOAP outperform their non-KL counterparts by learning the preconditioners compositionally, so that each stage corrects what remains after the last. Available in HeavyBall 3.1.1, with major PSGD stability backports.
1
4
47
3,102
1
2
83
also, short sided KL-Shampoo = short-sided Shampoo^2 = Muon
1
1
3
116
1
1
2
51
KL-Shampoo
1
1
4
92
Shampoo^2
1
1
4
100
Wu Lin retweeted
Wu Lin@LinYorkerApr 19
Replying to @weijie444
@weijie444 Looks like a KFAC-based method with modern clipping? G(ZZ^T)^{-1} is known as the FOOF update arxiv.org/abs/2201.12250 while msgn() can be interpreted as "generalized (preconditioned) gradient norm clipping" arxiv.org/abs/2506.01913 .

Gradient Descent on Neurons and its Link to Approximate...

Second-order optimizers are thought to hold the potential to speed up neural network training, but due to the enormous size of the curvature matrix, they typically require approximations to be...

arxiv.org
Weijie Su
@weijie444Apr 16
We released "The Newton--Muon Optimizer" . We show that Muon is secretly an implicit Newton method, and use this insight to build a better one. 1/n Paper: arxiv.org/abs/2604.01472

The Newton-Muon Optimizer

The Muon optimizer has received considerable attention for its strong performance in training large language models, yet the design principle behind its matrix-gradient orthogonalization remains...

arxiv.org
2
6
53
6,405
This work builds on my ICML 2019 paper (with @MarkSchmidtUBC and @EmtiyazKhan), extending a variational Bayes-based geometric framework to modern NN optimization. It can be used to design methods for Bayesian inference, numerical optimization, and gradient-free optimization.
1
3
126
In the matrix-valued case, we also consider a matrix variant of Adafactor, which can be viewed as a structural version of RMSProp using the von Neumann divergence. KL-Shampoo also outperforms this matrix Adafactor.
1
1
2
58
Surprisingly, KL-Shampoo no longer requires the empirical heuristics used in Shampoo and SOAP—such as step-size grafting and running Adam in the eigenbasis. Empirically, KL-Shampoo boosts Shampoo’s performance and outperforms SOAP while matching SOAP’s per-iteration runtime.
1
1
2
74
Because Shampoo and SOAP use Frobenius-norm–based estimation while quasi-Newton methods have transitioned to KL-based schemes such as BFGS and DFP, we are motivated to adopt a KL-based perspective.
1
1
2
78

No more items