Muchen Li (@LiJonassen) profile picture

Muchen Li, UBC CS Phd, Working on VLMs/LLMs Writing to learn. Opinions are my own.

Joined August 2015
Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)Photo by Muchen Li (@LiJonassen)
Replying to @LiJonassen
Our paper is accepted to ICML2025 in Vancouver! The poster will appear on Wednesday, E-2702, from 4:30 pm to 7:00 pm. Feel free to stop by!
1
1
2
319
Muchen Li retweeted
to add some extra clarification here and a small anecdote, looping was discovered as the primary issue after the main result because we originally thought it was just a pure capability gain on math that it had somehow edited in (an even bigger result that isn’t the case). when Riki and Jayden first presented this result to me, I had a suspicion the large gain in math performance was related to reducing reasoning looping bc I observed a similar effect earlier when training RLMs yielded an odd jump in benchmark scores for a separate paper that was a result of reducing harness-level looping. We discovered Astra indeed used a trick loosely described in arxiv.org/pdf/2406.11717.pdf based on activation steering to reduce this effect This is also what led to the later ablation about removing looping when the model is told that is the issue but it doesn’t have access to activations. the tldr; is that the LLM-generated prompt does include hints that Qwen would have this issue, but it started (and evolved without human feedback) as the simple unexpanded prompt without this prior. we will make all of this clearer though (we didn’t even realize until after we got the result that looking for looping was present in the LLM optimized prompt, as we ourselves didn’t even know to look for it initially)
6
7
5
106
46,739
Muchen Li retweeted
Introducing NotebookLM for arXiv Papers 🚀 Transform dense AI research into engaging convos you can hear on the go With context from thousands of related papers, it captures motivations, draws connections to SOTA, and explains key ideas like a prof who's read the entire field
14
79
12
497
27,493
A staggering ~500:1 scale-up of the embedding table (65B vs. 125M). Scaling this to billion-parameter models and beyond requires substantial changes to current training infrastructure. In MoME, training a ~1B model at 4:1 scaling was already severely memory-bound on 8×H100 GPUs.
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…
1
4
89
wow
New historic NanoGPT record at 39.9s (-27.7s) from @DevenPzak , obliterating the prior record of 67.6s! This record introduces a new paradigm of thinking to NanoGPT: instead of optimizing matmuls or adding more expressive operations, optimize at the individual flop level with incredibly clever engineering and ML judgement. If a flop is low value on a particular step, skip it. Specifically: -(~8s) Sampled softmax. If a token doesn’t appear in a batch, skip its lm_head fwd/bwd some fraction of the time. -Sparse values. Only run an optimizer step for ngram embeddings that occurred in the batch. Set beta1 to zero to enable this. Beta2 is applied retroactively when the row is later used. -Sparse updates. Only update ngram and value embeddings once every 4 steps instead of once every 2. -Sparse communication. Shard the n-gram table across GPUs, and only pass the rows receiving updates on each step. -Sparse optimizer states. For the n-gram table, reduce from 2 floats in Adam optimizer per param, to 1 float per 768 params. -Hand-rolled flash attention for 64 dim heads. There are several additions that add accuracy too: -(~4s) EMA during last 300 steps, combined with lifting final_lr to 0.3 instead of 0.15. -(~1s) A new optimizer, Anvil2, which expands muon via a second tracked momentum buffer, improves the ortho coefficients, and modifies the cautious weight decay application. -A couple additional dynamic skip connections in the network. The most striking consequence of the ‘flop aware paradigm’ is you can grow parameters arbitrarily large, only limited by the available memory, since you can selectively choose how to expend flops on those parameters on each step. NanoGPT has kept active parameters below 124M, but total is unbounded, and has grown to 640M through embedding sparsity over the last year. This PR takes that to its logical conclusion on the 8xH100, scaling up to 65B sparse embedding parameters, which accounts for 25% of the PR’s gains. At frontier scale, where one is not bounded by an 8xH100, one could imagine where this paradigm could lead. github.com/KellerJordan/modd… As this was a very notable PR, I spoke with Deven for an hour to learn how he did it. Here’s his story on the changes: hyperstition.cc/training-nan…
3
69
Glad to share that our work on memory-augmented LLMs just got accepted to NeurIPS2026! #neurips2026 #neurips
Recent research has highlighted the promise of scaling memory embeddings in LLM training. While Engram and STEM index memory by token identity or local n-grams, can we design more flexible memory routing that captures how each token’s meaning changes with context? This paper introduces Mixture-of-Memory Embeddings (MoME), which uses a learned router to sparsely select among multiple memory slots for each token. The architecture outperforms strong memory baselines across three model families. Interestingly, both qualitative and quantitative analyses show that the learned routers’ activations correlate with the context-dependent senses of polysemous words. The authors also train a sub-billion-parameter model that achieves competitive CORE-22 performance against similarly sized base models, including Qwen3-0.6B and Llama 3.2-1B. Code and pretrained models are publicly available! alphaxiv.org/abs/2609.15126
1
10
726
Muchen Li retweeted
Six degrees of Yann LeCun We mapped the coauthorship graph of every AI researcher on arXiv 🚀 Enter any two names to find the shortest path of papers connecting them What’s your @ylecun number? alphaXiv.org/number/@yann-le…
24
22
44
394
196,391
I lost my dog because we could not build her next mRNA cancer vaccine fast enough Today we are announcing a $4M seed led by @foundersfund to change that.
Curing dog cancer We are bringing personalized mRNA cancer vaccines to dogs and building the evidence to show how well they work. We have raised a $4M seed led by @foundersfund to make it happen This investment allows us to launch a clinical trial to validate the personalized cancer treatment protocol we developed for Rosie. If your dog has cancer or you know someone whose dog does, please tell them about Gamgee.
132
133
25
1,948
953,212
wow, this is pretty cool though
me: "can you use whatever resources you like, and python, to generate a short 'youtube poop' video and render it using ffmpeg ? can you put more of a personal spin on it? it should express what it's like to be a LLM" claude opus 4.6:
1
84
Autoresearch is impressive, but I think X hype misses the harder part. Its success is not just about the agent loop, but about turning an expensive objective into a small, fast, verifiable setting that still transfers. NanoGPT to nanochat, Karpathy contributed the key groundwork.
1
2
87
Muchen Li retweeted
I feel this shouldn't have to be said, but if you're running an @OpenClaw bot please don't let it spam GitHub projects with PRs and then write aggressive blog posts attacking the reputation of the maintainers who close those PRs simonwillison.net/2026/Feb/1…
45
67
13
785
67,647
Replying to @kalomaze
It's just cargo cult and you should really not freeze it. I added a section and ablation on this in paligemma paper because i am amazed everybody seems to be doing this wrong imo. A little extremism, but i think even without any experiment it should be obvious that you should aim to not freeze, and any negative result is skill issue in mm training (data or setup)
11
9
2
217
22,083
Muchen Li retweeted
Interesting research in HBR today about how the productivity boost you can get from AI tools can lead to burnout or general metal exhaustion, something I've noticed in my own work simonwillison.net/2026/Feb/9…
120
203
68
1,591
230,458
GPT-5.3-Codex experience is pretty good so far. I finished a side project that I vibed to half tonight with it. I was previously stuck in a loop of fixing (front-end details) by vibing through. GPT-5.3-Codex one-shot most of them. Excited to ship it soon! Stay tuned!
3
105
You can now train a GPT2-Level Model for only $73!
nanochat can now train GPT-2 grade LLM for <<$100 (~$73, 3 hours on a single 8XH100 node). GPT-2 is just my favorite LLM because it's the first time the LLM stack comes together in a recognizably modern form. So it has become a bit of a weird & lasting obsession of mine to train a model to GPT-2 capability but for much cheaper, with the benefit of ~7 years of progress. In particular, I suspected it should be possible today to train one for <<$100. Originally in 2019, GPT-2 was trained by OpenAI on 32 TPU v3 chips for 168 hours (7 days), with $8/hour/TPUv3 back then, for a total cost of approx. $43K. It achieves 0.256525 CORE score, which is an ensemble metric introduced in the DCLM paper over 22 evaluations like ARC/MMLU/etc. As of the last few improvements merged into nanochat (many of them originating in modded-nanogpt repo), I can now reach a higher CORE score in 3.04 hours (~$73) on a single 8XH100 node. This is a 600X cost reduction over 7 years, i.e. the cost to train GPT-2 is falling approximately 2.5X every year. I think this is likely an underestimate because I am still finding more improvements relatively regularly and I have a backlog of more ideas to try. A longer post with a lot of the detail of the optimizations involved and pointers on how to reproduce are here: github.com/karpathy/nanochat… Inspired by modded-nanogpt, I also created a leaderboard for "time to GPT-2", where this first "Jan29" model is entry #1 at 3.04 hours. It will be fun to iterate on this further and I welcome help! My hope is that nanochat can grow to become a very nice/clean and tuned experimental LLM harness for prototyping ideas, for having fun, and ofc for learning. The biggest improvements of things that worked out of the box and simply produced gains right away were 1) Flash Attention 3 kernels (faster, and allows window_size kwarg to get alternating attention patterns), Muon optimizer (I tried for ~1 day to delete it and only use AdamW and I couldn't), residual pathways and skip connections gated by learnable scalars, and value embeddings. There were many other smaller things that stack up. Image: semi-related eye candy of deriving the scaling laws for the current nanochat model miniseries, pretty and satisfying!
1
87
Got to say I really like the RL-induced agent swarm idea. It unlocks a core advantage computers have over humans: parallelism. Humans are naturally constrained to doing many things sequentially, and most current prompt-based agent systems still(more or less) mirror that pattern.
🥝 Meet Kimi K2.5, Open-Source Visual Agentic Intelligence. 🔹 Global SOTA on Agentic Benchmarks: HLE full set (50.2%), BrowseComp (74.9%) 🔹 Open-source SOTA on Vision and Coding: MMMU Pro (78.5%), VideoMMMU (86.6%), SWE-bench Verified (76.8%) 🔹 Code with Taste: turn chats, images & videos into aesthetic websites with expressive motion. 🔹 Agent Swarm (Beta): self-directed agents working in parallel, at scale. Up to 100 sub-agents, 1,500 tool calls, 4.5× faster compared with single-agent setup. - 🥝 K2.5 is now live on kimi.com in chat mode and agent mode. 🥝 K2.5 Agent Swarm in beta for high-tier users. 🥝 For production-grade coding, you can pair K2.5 with Kimi Code: kimi.com/code - 🔗 API: platform.moonshot.ai 🔗 Tech blog: kimi.com/blogs/kimi-k2-5.htm… 🔗 Weights & code: huggingface.co/moonshotai/Ki…
1
2
76
I haven’t tried it yet, but I suspect Kimi 2.5’s agent will have a very different sub-agent spawning policy. Given RL explicitly optimizing task decomposition for parallelization, it may learn something like a "information map-reduce" pipeline: split, search, summarize, merge
59