Moving humans forward through radical AI research

Melbourne, Australia
Joined May 2026
Muon has been a recent obsession of ours, so we dug deeper. It changes the loss curve, but does it change what a model learns? We trained matched AdamW and Muon GPT-2-class models, held validation loss fixed, and compared their SAE features by firing patterns over the same 1M tokens. The result: Muon model’s features match an AdamW model’s features about as well as two AdamW seeds match each other. So changing the optimizer perturbs SAE feature identity about as much as changing the random seed. Muon’s advantage is not “different features.” It is “same features, different geometry.”
3
1
393