Researcher at NVIDIA // ex-Researcher at Microsoft, PhD from MIT EECS
Cambridge, MA
Joined February 2020
- Tweets73
- Following316
- Followers739
- Likes92
Kwangjun Ahn retweeted
sacled up to 12.7B dense, 5.5T tokens.
- polynorm (optimized kernel)
- grouped diff attn (their work)
- parallel muonclip (adopt alltoall like mainhorse, essential, dion)
- 80M batch
it's still non-reasoning, also not moe though... keep pushing guys!
arxiv.org/abs/2511.07464
they also dropped fsdp2 optimized muon. though they don't use muon for 2.6b dense model, i think it's just beginning and they are preparing larger one. they pipeline muon's comm-comp with calc flops and the code is neat. not sure if it's existing method.
huggingface.co/Motif-Technol…
New improvement in Dion leads to a speedup that makes orthonormal updates (eg. Muon) more scalable for larger matrices. The trick: carefully using Newton-Schulz (on smaller matrices) as Dion's backend. Updates to our microsoft/dion codebase are coming soon---stay tuned!
Kwangjun Ahn retweeted
Join us on Sept 24 at 8 AM PT for Microsoft Research Forum Season 2 – a virtual series highlighting purposeful research and its real-world impact, from fundamental exploration to advancing AI responsibly, scaling innovation through products and open source, and driving positive change for society. Register now: msft.it/6011scy27
Kwangjun Ahn retweeted
love the repo! clean code, good practices but still not overly over-engineered, triton kernels, well documented, simple reference implementations alongside optimized code. nice
Kwangjun Ahn retweeted
Looks like extremely exciting and useful work by @KwangjunA, Byron Xu, Natalie Abreu, @JohnCLangford and @GagMagakyan
github.com/microsoft/dion/
(2/2)
Kwangjun Ahn retweeted
I had wondered why there was no official Dion implementation by the authors... I guess now we know. This repository looks dope: FSDP Muon and Dion implementations, triton kernels for Newton-Schulz, and lots of practical advice
(1/2)
Kwangjun Ahn retweeted
[1/6] Curious about Muon, but not sure where to start? I wrote a 3-part blog series called “Understanding Muon” designed to get you up to speed—with The Matrix references, annotated source code, and thoughts on where Muon might be going.
Kwangjun Ahn retweeted
Apparently Dion is now being worked on for Torch Titan: github.com/pytorch/torchtita… :-)
Kwangjun Ahn retweeted
Since nobody asked :-), here is my list of papers not to be missed from ICML:
1) Dion: distributed orthonormalized updates (well, technically not at ICML, but everyone's talking about it).
2) MARS: Unleashing the Power of Variance Reduction for Training Large Models
3) ...
Kwangjun Ahn retweeted
But actually this is the og way of doing it and should stop by E-2103 to see @jxbz and Laker Newhouse whiteboard the whole paper.
Kwangjun Ahn retweeted
Schedule-Free methods, which forgo cosine/linear schedulers by averaging iterates and computing gradients at interpolated points, yield smoother training curves. It's still unclear why they work well, and this paper explains the phenomenon through the river-valley loss landscape.
Kwangjun Ahn retweeted
If you are at ICML 2025, come check out our oral presentation about the non-convex theory of Schedule Free SGD in the Optimization session tomorrow!
This work was done with amazing collaborators @KwangjunA and @AshokCutkosky.
ICML: come check out our Oral Presentation on Schedule-free training theory based on an elegant online learning!
ICLR:
@edward_s_hu and I will be presenting our work "The Belief State Transformer" at the 1st poster session. (#269) Please come check it out! (github: github.com/microsoft/BST)
Kwangjun Ahn retweeted
The Belief State Transformer edwardshu.com/bst-website/ is at ICLR this week. The BST objective efficiently creates compact belief states: summaries of the past sufficient for all future predictions. See the short talk: microsoft.com/en-us/research… and @mgostIH for further discussion.
Kwangjun Ahn retweeted
New reqs for low to high level researcher positions:
jobs.careers.microsoft.com/g… ,
jobs.careers.microsoft.com/g…,
jobs.careers.microsoft.com/g…,
jobs.careers.microsoft.com/g…,
with postdocs from Akshay and @MiroDudik xitter.cf/MiroDudik/status/18367… . Please apply or pass to those who may :-)
Kwangjun Ahn retweeted
Last year, we had offers accepted from @KwangjunA, @riashatislam, @Tea_Pearce , @pratyusha_PS while Akshay and @MiroDudik hired 7(!) postdocs.
New reqs for low to high level researcher positions:
jobs.careers.microsoft.com/g… ,
jobs.careers.microsoft.com/g…,
jobs.careers.microsoft.com/g…,
jobs.careers.microsoft.com/g…,
with postdocs from Akshay and @MiroDudik xitter.cf/MiroDudik/status/18367… . Please apply or pass to those who may :-)