🧪 @thinkymachines ✍️ anon feedback @ https://xitter.cf/t.co/RIhBhjMRdD

🌉
Joined January 2010
I just wrote my first blog post in four years! It is called "Deriving Muon". It covers the theory that led to Muon and how, for me, Muon is a meaningful example of theory leading practice in deep learning (1/11)
13
140
10
1,043
128,498
Jeremy Bernstein retweeted
we’re hiring a founding model designer to shape how Inkling behaves and feels. bring a writer’s precision, a designer’s judgment, and the technical rigor to translate your ideas into data and evals. please DM your best work or apply below. jobs.ashbyhq.com/ThinkingMac…
1
14
2
68
9,380
We are making lots of improvements to Tinker. Now we have a new optimizer to try out!
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback! See our optimizers guide at tinker-docs.thinkingmachines…
4
15
1
178
9,749
Jeremy Bernstein retweeted
We're hiring safety researchers at @thinkymachines! 🦺 My team works on safety across the model development stack -- pre-training data filtering, harmful capability evals, safety post-training, red-teaming, abliteration/malicious fine-tuning. We're particularly interested in building evals and tooling to make strong safety cases for open-weights releases. We think this is where some of the most important safety research will happen in the next few years. If any of this resonates, apply here: jobs.ashbyhq.com/ThinkingMac…
16
52
6
581
88,454
Jeremy Bernstein retweeted
Inkling-Small is comparable to Inkling at a quarter the size. Weights are open, fine-tunable on Tinker today. Look forward to seeing what people make with it.
Today, we are releasing Inkling-Small. Inkling-Small achieves comparable performance to Inkling at a quarter of its size. It features 276B total parameters, 12B active. We are making the full weights available. thinkingmachines.ai/news/ink… Fine-tune it on Tinker today, or chat with it in text, image, and audio on Tinker Playground.
148
283
36
3,718
478,207
our first Inkling
Today, we are introducing Inkling. Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available. thinkingmachines.ai/news/int… Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵
4
3
1
109
5,078
Jeremy Bernstein retweeted
Announcing a new division of Midjourney called "Midjourney Medical"
2,475
5,028
4,086
39,497
18,904,014
Working on pretraining, it often feels like we are building an intricate physical system complete with dynamical laws and dimensional scaling. I did not expect that using our models would also start to feel like interacting with a dynamical system (1/2) xitter.cf/thinkymachines/status/…
People talk, listen, watch, think, and collaborate at the same time, in real time. We've designed an AI that works with people the same way. We share our approach, early results, and a quick look at our model in action. thinkingmachines.ai/blog/int…
4
10
258
26,813
Interaction models continuously absorb information from the world and think and respond in real time. They are aware of the passage of time, know when to listen and can interrupt based on audio or visual cues. I am so excited by all the work being done here at Thinky (2/2)
1
2
1
68
2,619
Jeremy Bernstein retweeted
worth a read for any muon enjoyers
Introducing Aurora, a new optimizer for training frontier-scale models. We train Aurora-1.1B, which achieves 100x data efficiency on open-source internet data. Despite having 25% fewer parameters, 2 orders of magnitude fewer training tokens, and using fully open-source internet-only data, Aurora matches Qwen3-1.7B on several benchmarks. Aurora was developed after identifying a major failure mode that can occur under Muon, an increasingly popular optimizer that has shown strong gains over Adam(W). We find that Muon can cause a huge percentage of neurons to effectively die early in training, reducing effective network capacity so that many parameters no longer meaningfully contribute to network outputs. By redistributing update energy more uniformly across neurons while preserving Muon’s stability properties, Aurora prevents neuron death and recovers substantial model capacity. What makes this work especially exciting is that it points toward a broader direction for ML research: better optimizers may not come purely from elegant mathematical abstractions, but from understanding and addressing the concrete dynamics and pathologies that emerge inside real training systems.
1
10
145
20,011
I found this to be a wonderful paper. I made a related observation last year: the hparam scalings for learning rate and batch size in our signSGD paper (from 2018!) have been found to be compute optimal in recent LLM scaling studies (1/4)
Optimization theory for adaptive methods actually predicts most of what we know about hyperparameter scaling in LLM pretraining, and suggests new strategies as well. We did a deep dive here.
3
13
1
224
30,972
There is much, much more in @orvieto_antonio and team's paper and I highly recommend reading it. While for any theoretical optimization result it is important to scrutinize the assumptions, I do find it striking that these theoretical scalings seem to match practice (3/4)
1
2
16
2,087
These scalings are usually derived using SDEs or other means. Like the authors above, I am excited by the idea that we can build an optimization-theoretic foundation for neural network training Tagging my old collaborator @yuxiangw_cs who I think would find this cool! (4/4)
1
2
19
1,567
Jeremy Bernstein retweeted
Muon can accelerate LLM training, but does that benefit transfer to regulatory DNA sequence modeling with its different data distribution? 🧬 Our results show that Muon with independent weight decay (MuonW) hits our validation perplexity target in ~37% fewer FLOPs than the best Adam configuration.
5
25
3
131
13,080
Jeremy Bernstein retweeted
To support open and collaborative science, we offer Tinker grants for researchers advancing the field. This week we’re featuring publications by some of our early research grant recipients! thinkingmachines.ai/blog/tin…
3
37
1
256
154,963
Jeremy Bernstein retweeted
1/14 Is Muon “better” than Shampoo? We argue that their relationship parallels Adam's relationship with Signum. Analogous to @lukas_balles and Hennig’s (2018) decomposition of Adam into element-wise scaled Signum, we can decompose Shampoo as left- and right-adapted Muon.
3
45
6
265
33,110
Co-teaching this class was one of the highlights of my time in @phillip_isola's group at MIT and something that I am proud to have been a part of! xitter.cf/phillip_isola/status/2…
Our grad-level "Deep Learning" course (MIT's 6.7960) is now freely available online through OpenCourseWare: ocw.mit.edu/courses/6-7960-d… Lecture videos, psets, and readings are all provided. Had a lot of fun teaching this with @sarameghanbeery and @jxbz!
2
9
1
110
9,723
I also find it amusing that a couple homework problems I wrote are now freely available as part of MIT OpenCourseWare. xitter.cf/sprooos/status/1710748…
from my homework in MIT 6.s898 this week
1
1
25
3,130
Jeremy Bernstein retweeted
Our grad-level "Deep Learning" course (MIT's 6.7960) is now freely available online through OpenCourseWare: ocw.mit.edu/courses/6-7960-d… Lecture videos, psets, and readings are all provided. Had a lot of fun teaching this with @sarameghanbeery and @jxbz!
17
276
9
1,740
142,940