Kilian Haefeli retweeted
1/5
New blog: How to Build a Scientific Scaling Ladder.
I've spent the past three months studying pretraining and scaling. This guide brings together public research on using small runs to predict large-model training configurations and performance.
jiaxuanzou0714.github.io/en/…
Kilian Haefeli retweeted
ClusterMAX 3.0: The Industry Standard GPU Cloud Rating System Returns
In gory detail: reliability, performance, support, pricing
—and, of course, security—
in our most thorough analysis of GPU cloud providers globally
semianalysis.substack.com/p/…
Kilian Haefeli retweeted
Dylan Patel (@dylan522p) of @SemiAnalysis_ says open source is dying:
"There's multiple Chinese model labs who are telling all the inference guys, 'Our next model's not going to be open source. We're going to license it to you.'"
" Open is dying quickly, unfortunately."
Readers added context they thought people might want to know
This July 2026 quote (recirculated without date) claimed Chinese labs were shifting away from open source. Multiple labs have since released large open-weight models, including Qwen updates in Aug-Sept 2026.
huggingface.co/blog/state-of-…
rits.shanghai.nyu.edu/news/
wired.com/story/chinas-o…
x.com/sourceryy/stat…
Kilian Haefeli retweeted
I spent a humiliating amount of time learning how to make animated graphs, just to illustrate a fairly obvious point.
“Forecasting s-curves is hard”
My views on why carefully following daily figures is unlikely to provide insight.
constancecrozier.com/2020/04…
Kilian Haefeli retweeted
1/ If you have been reading about discrete diffusion, you might have asked yourself the following question:
What does a discrete diffusion model actually learn?
A denoiser? A concrete score? A bridge plug-in?
The literature treats them as interchangeable. They are not! 🧵
Kilian Haefeli retweeted
How to scale the batch size in LLM pretraining?
New paper on a scaling law that splits token budget D into training steps and batch size.
💺 arxiv.org/abs/2607.01487
Kilian Haefeli retweeted
just your friendly reminder to throw away any RL paper that only tests their method on Qwen models :)
Kilian Haefeli retweeted
Ever wondered how Speculative Decoding interacts with production MoE models?
Conventional wisdom: MoE + speculative decoding = too many experts to load, gains disappear.
Reality: MoE amplifies speculative decoding.
Checkout Cohere Blogpost: cohere.com/blog/mixture-of-e…
Kilian Haefeli retweeted
I work on the JAX team. If you're new to the field ignore this bait. The things you should focus on are understanding the math and how to program accelerators - really master your hardware and your methods. We try to make a great tool but don't obsess over tools early.
Kilian Haefeli retweeted
🤯 big update to our flow map language models paper!
we believe this is the future of non-autoregressive text generation.
read about it in the blog: one-step-lm.github.io/blog/
full details in the paper: arxiv.org/abs/2602.16813
we introduce a new class of continuous flow-based language models and distill them into their corresponding flow map for one-step text generation.
we beat all discrete diffusion baselines at ~8x speed!
v2 gives a complete theory of the flow map over discrete data, with three equivalent ways to learn it (semigroup, lagrangian, eulerian). it turns out you can train these with cross-entropy objectives that look very similar to standard discrete diffusion — but without the factorization error that kills discrete methods at few steps.
beyond improving results across the board, we showcase properties that are unique to continuous flows. in particular, inference-time steering and guidance become straightforward. autoguidance brings generative perplexity down to 51.6 on LM1B, while discrete baselines completely collapse at the same guidance scale.
we also show reward-guided generation for steering topic, sentiment, grammaticality, and safety at inference time — and it works even at 1-2 steps with our flow map model. simple, well-understood techniques from continuous flows just work incredibly well in practice for language.
we’re extremely excited about the future of this class of models.
stay tuned for results on scaling, reasoning, and reinforcement learning-based fine-tuning. 🚀
Awesome deepdive by kimbo!!
Previously when writing the NVIDIA Tensor Core evolution article, I realized how scarce the info about Blackwell micro architecture is
So I set off doing this to bridge the gap.
Super thankful for my colleagues and collaborators
Thank you @snowclipsed for reviewing too
Kilian Haefeli retweeted
Join us at @CVPR in Denver for a full-day tutorial about Analytic Understanding of Diffusion Models.
The training objective of diffusion models has a closed-form solution -- yet it only memorizes. How do real models generalize? We'll unpack this paradox and the emerging analytical theory behind it.
@yuancy @CScarvelis @MasonKamb @WangBinxu @vincesitzmann @JustinMSolomon @SuryaGanguli



























