
🧪 @thinkymachines ✍️ anon feedback @ https://xitter.cf/t.co/RIhBhjMRdD
🌉
Joined January 2010
- Tweets1.1K
- Following708
- Followers8K
- Likes3K
Pinned Tweet
I just wrote my first blog post in four years! It is called "Deriving Muon". It covers the theory that led to Muon and how, for me, Muon is a meaningful example of theory leading practice in deep learning
(1/11)
Jeremy Bernstein retweeted
we’re hiring a founding model designer to shape how Inkling behaves and feels.
bring a writer’s precision, a designer’s judgment, and the technical rigor to translate your ideas into data and evals.
please DM your best work or apply below.
jobs.ashbyhq.com/ThinkingMac…
We are making lots of improvements to Tinker. Now we have a new optimizer to try out!
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at tinker-docs.thinkingmachines…
Jeremy Bernstein retweeted
We're hiring safety researchers at @thinkymachines! 🦺
My team works on safety across the model development stack -- pre-training data filtering, harmful capability evals, safety post-training, red-teaming, abliteration/malicious fine-tuning.
We're particularly interested in building evals and tooling to make strong safety cases for open-weights releases. We think this is where some of the most important safety research will happen in the next few years.
If any of this resonates, apply here: jobs.ashbyhq.com/ThinkingMac…
Jeremy Bernstein retweeted
Inkling-Small is comparable to Inkling at a quarter the size. Weights are open, fine-tunable on Tinker today. Look forward to seeing what people make with it.
Today, we are releasing Inkling-Small.
Inkling-Small achieves comparable performance to Inkling at a quarter of its size. It features 276B total parameters, 12B active. We are making the full weights available.
thinkingmachines.ai/news/ink…
Fine-tune it on Tinker today, or chat with it in text, image, and audio on Tinker Playground.
our first Inkling
Today, we are introducing Inkling.
Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.
thinkingmachines.ai/news/int…
Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. đź§µ
Working on pretraining, it often feels like we are building an intricate physical system complete with dynamical laws and dimensional scaling. I did not expect that using our models would also start to feel like interacting with a dynamical system
(1/2)
xitter.cf/thinkymachines/status/…
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way.
We share our approach, early results, and a quick look at our model in action.
thinkingmachines.ai/blog/int…
Jeremy Bernstein retweeted
worth a read for any muon enjoyers
Introducing Aurora, a new optimizer for training frontier-scale models.
We train Aurora-1.1B, which achieves 100x data efficiency on open-source internet data. Despite having 25% fewer parameters, 2 orders of magnitude fewer training tokens, and using fully open-source internet-only data, Aurora matches Qwen3-1.7B on several benchmarks.
Aurora was developed after identifying a major failure mode that can occur under Muon, an increasingly popular optimizer that has shown strong gains over Adam(W). We find that Muon can cause a huge percentage of neurons to effectively die early in training, reducing effective network capacity so that many parameters no longer meaningfully contribute to network outputs.
By redistributing update energy more uniformly across neurons while preserving Muon’s stability properties, Aurora prevents neuron death and recovers substantial model capacity.
What makes this work especially exciting is that it points toward a broader direction for ML research: better optimizers may not come purely from elegant mathematical abstractions, but from understanding and addressing the concrete dynamics and pathologies that emerge inside real training systems.
I found this to be a wonderful paper.
I made a related observation last year: the hparam scalings for learning rate and batch size in our signSGD paper (from 2018!) have been found to be compute optimal in recent LLM scaling studies
(1/4)
There is much, much more in @orvieto_antonio and team's paper and I highly recommend reading it.
While for any theoretical optimization result it is important to scrutinize the assumptions, I do find it striking that these theoretical scalings seem to match practice
(3/4)
These scalings are usually derived using SDEs or other means. Like the authors above, I am excited by the idea that we can build an optimization-theoretic foundation for neural network training
Tagging my old collaborator @yuxiangw_cs who I think would find this cool!
(4/4)
Jeremy Bernstein retweeted
Muon can accelerate LLM training, but does that benefit transfer to regulatory DNA sequence modeling with its different data distribution? 🧬
Our results show that Muon with independent weight decay (MuonW) hits our validation perplexity target in ~37% fewer FLOPs than the best Adam configuration.
Jeremy Bernstein retweeted
To support open and collaborative science, we offer Tinker grants for researchers advancing the field.
This week we’re featuring publications by some of our early research grant recipients!
thinkingmachines.ai/blog/tin…
Jeremy Bernstein retweeted
1/14 Is Muon “better” than Shampoo?
We argue that their relationship parallels Adam's relationship with Signum. Analogous to @lukas_balles and Hennig’s (2018) decomposition of Adam into element-wise scaled Signum, we can decompose Shampoo as left- and right-adapted Muon.
Co-teaching this class was one of the highlights of my time in @phillip_isola's group at MIT and something that I am proud to have been a part of!
xitter.cf/phillip_isola/status/2…
Our grad-level "Deep Learning" course (MIT's 6.7960) is now freely available online through OpenCourseWare: ocw.mit.edu/courses/6-7960-d…
Lecture videos, psets, and readings are all provided.
Had a lot of fun teaching this with @sarameghanbeery and @jxbz!
I also find it amusing that a couple homework problems I wrote are now freely available as part of MIT OpenCourseWare.
xitter.cf/sprooos/status/1710748…
Check out homework 2 if you want to sharpen your understanding of hyperparameter transfer / spectral condition / steepest descent / Muon!
ocw.mit.edu/courses/6-7960-d…
Jeremy Bernstein retweeted
Our grad-level "Deep Learning" course (MIT's 6.7960) is now freely available online through OpenCourseWare: ocw.mit.edu/courses/6-7960-d…
Lecture videos, psets, and readings are all provided.
Had a lot of fun teaching this with @sarameghanbeery and @jxbz!

























