Stumbled upon this. @wen_kaiyue & co. of arxiv.org/abs/2606.16899 may want to know that \kappa(\rho) -> \rho as p/q deviates from 1, i.e. for the purpose of steady-state weight norm, Muon's msign is effectively L2 normalization for tall/wide matrices: xitter.cf/EIFY/status/2041004165…
ALT https://arxiv.org/abs/2606.16899 page 10 excerpt
So I finally get around to test this thoroughly with Gaussian random vector / matrix: github.com/EIFY/normalized_u…
The prediction works great for normalized vector update and Muon / Scion spectral ortho. update with n=4m matrix, but not so much for steady-state norm when n=m. Why? 🤔
Muon(Clip) used by GLM & Kimi has a RMS matching factor of 0.2 in order to reuse AdamW's optimal LR. My theory is that it actually makes Muon's momentum-dependent effective LR roughly match AdamW's (momentum-independent) LR: (1.95/0.05) ** .5 * 0.2 = 1.25
1/3
ALT Muon's effective LR
ALT MuonClip
So their eff. LR matching is in fact more accurate. I don't know if they got it thru trial & error or they had their own theory internally.
DS V4 tech report: huggingface.co/deepseek-ai/D…
My preprint (that started about WD): arxiv.org/abs/2512.08217
3/3
So I finally get around to test this thoroughly with Gaussian random vector / matrix: github.com/EIFY/normalized_u…
The prediction works great for normalized vector update and Muon / Scion spectral ortho. update with n=4m matrix, but not so much for steady-state norm when n=m. Why? 🤔
I happened to be running this ablation. Preliminary result based on training a ViT-S on ImageNet-1k for 90 epochs says it's better to leave the biases out 🤔
Background: A modified DeiT base has been the CV workhorse for Scion papers... 1/5
xitter.cf/Ji_Ha_Kim/status/20372…
2. For my ViT-S I made sure that QKV grad. are separately orthogonalized and the input dim. of patchifier are flattened. In the process I already left out the bias of QKV.
3. I incorporated @tmpethick's 1.0 init. mo for the unbiased exp.
4/5
If the hidden dimension D is a major bottleneck then we can expand it cheaply & exponentially in the last few layers by replacing residual connections with concatenation, tweaking the MLP shape, or activating more experts. Has anyone tried that?
xitter.cf/nthngdy/status/2032172…
TIL scipy has every possible inverse of the regularized incomplete beta function: scipy.special.{btdtria, btdtrib, betaincinv}
Convenient for working with Beta distribution!
Quite a late bloomer, (constrained) Scion.
Diff. from the DeiT exp. in the paper: It's based on "better baseline" ViT-S/16 (global avg. pooling, sincos2d etc.) and trained for 90 ep. Parameters are all reinitialized and it uses Polar Express for msign.
github.com/EIFY/mup-vit/tree…
ALT ImageNet-1k Top-1 val. accuracy after 90 epochs: Scion 78.47, AdamW 77.45, unScion 73.61
(4/4) So, spectral chisel can be as accurate as other methods except msign-based hardcapping. It may depend on how many singular values we need to cap but # of steps needed should be ~O(n), so the total cost is at most about that of a few matmuls. Repo: github.com/EIFY/spectral-chi…
P.S. I submitted a PR to fix that author name typo, lol
github.com/Arongil/lipschitz…
(1/4) Comparing such "spectral chisel" (1 iter/step) against the ground truth, direct hardcap (from @LakerNewhouse's repo), and msign-based hardcap (Newhouse et al., 2025) for capping n=4096 square unit gaussian random matrix at svdval <= 120:
xitter.cf/EIFY/status/1947062171…
ALT Hardcapping method comparison
(3/4) 🤨 Strangely the lipschitz-transformers repo doesn't implement the msign-based spectral hardcapping described in Appendix B. Instead it does this "direct hardcapping" attributed to @leloykun, presumably optimized with procedures similar to xitter.cf/YouJiacheng/status/189…
ALT Direct hardcapping
ALT Direct hardcapping vs. the ground truth, visualized
Improved stability against noise, idea of the noise-stability loss is borrowed from @leloykun 's code.
