pretraining @cohere & master @ETH

Zurich, Switzerland
Joined August 2012
Kilian Haefeli retweeted
Me and bro looking at PTX memory model
6
3
56
4,053
Kilian Haefeli retweeted
1/5 New blog: How to Build a Scientific Scaling Ladder. I've spent the past three months studying pretraining and scaling. This guide brings together public research on using small runs to predict large-model training configurations and performance. jiaxuanzou0714.github.io/en/…
7
65
4
572
26,655
Kilian Haefeli retweeted
rumours of backprop's demise are greatly exaggerated
We've figured out how to *pretrain transformers* with zeroth-order optimization and no backprop. Many of the core assumptions in optimization research are completely wrong. (paper out soon)
8
11
1
346
31,250
Kilian Haefeli retweeted
ClusterMAX 3.0: The Industry Standard GPU Cloud Rating System Returns In gory detail: reliability, performance, support, pricing —and, of course, security— in our most thorough analysis of GPU cloud providers globally semianalysis.substack.com/p/…
37
17
29
140
138,705
Kilian Haefeli retweeted
Dylan Patel (@dylan522p) of @SemiAnalysis_ says open source is dying: "There's multiple Chinese model labs who are telling all the inference guys, 'Our next model's not going to be open source. We're going to license it to you.'" " Open is dying quickly, unfortunately."
Readers added context they thought people might want to know
This July 2026 quote (recirculated without date) claimed Chinese labs were shifting away from open source. Multiple labs have since released large open-weight models, including Qwen updates in Aug-Sept 2026. huggingface.co/blog/state-of-… rits.shanghai.nyu.edu/news/ wired.com/story/chinas-o… x.com/sourceryy/stat…
13
14
3
439
76,945
I spent a humiliating amount of time learning how to make animated graphs, just to illustrate a fairly obvious point. “Forecasting s-curves is hard” My views on why carefully following daily figures is unlikely to provide insight. constancecrozier.com/2020/04…
256
3,002
568
10,591
Kilian Haefeli retweeted
5
10
1
447
16,094
Kilian Haefeli retweeted
Football won
24,433
419,318
20,819
1,951,089
69,723,089
1/ If you have been reading about discrete diffusion, you might have asked yourself the following question: What does a discrete diffusion model actually learn? A denoiser? A concrete score? A bridge plug-in? The literature treats them as interchangeable. They are not! 🧵
3
20
4
92
23,356
Kilian Haefeli retweeted
How to scale the batch size in LLM pretraining? New paper on a scaling law that splits token budget D into training steps and batch size. 💺 arxiv.org/abs/2607.01487
4
14
102
7,855
Kilian Haefeli retweeted
You know what model won’t reject your requests? Cohere North Mini Code
1
3
23
1,602
Kilian Haefeli retweeted
Introducing Cohere's first open-source coding model: North Mini Code Small & efficient, designed for agentic performance and built for community input.
68
263
101
2,268
600,930
Kilian Haefeli retweeted
To hell with big TMA long live ld/st
3
4
38
3,496
Kilian Haefeli retweeted
Introducing: Cohere Command A+ We’ve created our most powerful LLM yet, optimized it to run on as little hardware as possible, and released it open-source for all.
103
361
113
2,601
745,389
just your friendly reminder to throw away any RL paper that only tests their method on Qwen models :)
11
6
2
393
45,859
Kilian Haefeli retweeted
Ever wondered how Speculative Decoding interacts with production MoE models? Conventional wisdom: MoE + speculative decoding = too many experts to load, gains disappear. Reality: MoE amplifies speculative decoding. Checkout Cohere Blogpost: cohere.com/blog/mixture-of-e…
7
12
2
36
20,880
Kilian Haefeli retweeted
I work on the JAX team. If you're new to the field ignore this bait. The things you should focus on are understanding the math and how to program accelerators - really master your hardware and your methods. We try to make a great tool but don't obsess over tools early.
When looking at deep learning profiles, one of the most obvious tells between a mediocre and great candidate is whether they list PyTorch or JAX.
18
80
5
1,798
108,114
Kilian Haefeli retweeted
🤯 big update to our flow map language models paper! we believe this is the future of non-autoregressive text generation. read about it in the blog: one-step-lm.github.io/blog/ full details in the paper: arxiv.org/abs/2602.16813 we introduce a new class of continuous flow-based language models and distill them into their corresponding flow map for one-step text generation. we beat all discrete diffusion baselines at ~8x speed! v2 gives a complete theory of the flow map over discrete data, with three equivalent ways to learn it (semigroup, lagrangian, eulerian). it turns out you can train these with cross-entropy objectives that look very similar to standard discrete diffusion — but without the factorization error that kills discrete methods at few steps. beyond improving results across the board, we showcase properties that are unique to continuous flows. in particular, inference-time steering and guidance become straightforward. autoguidance brings generative perplexity down to 51.6 on LM1B, while discrete baselines completely collapse at the same guidance scale. we also show reward-guided generation for steering topic, sentiment, grammaticality, and safety at inference time — and it works even at 1-2 steps with our flow map model. simple, well-understood techniques from continuous flows just work incredibly well in practice for language. we’re extremely excited about the future of this class of models. stay tuned for results on scaling, reasoning, and reinforcement learning-based fine-tuning. 🚀
13
90
10
485
79,560
Awesome deepdive by kimbo!!
Previously when writing the NVIDIA Tensor Core evolution article, I realized how scarce the info about Blackwell micro architecture is So I set off doing this to bridge the gap. Super thankful for my colleagues and collaborators Thank you @snowclipsed for reviewing too
1
1
533
Kilian Haefeli retweeted
Join us at @CVPR in Denver for a full-day tutorial about Analytic Understanding of Diffusion Models. The training objective of diffusion models has a closed-form solution -- yet it only memorizes. How do real models generalize? We'll unpack this paradox and the emerging analytical theory behind it. @yuancy @CScarvelis @MasonKamb @WangBinxu @vincesitzmann @JustinMSolomon @SuryaGanguli
8
31
4
198
33,516