Assistant Professor @ Tsinghua University
Joined June 2016
- Tweets90
- Following745
- Followers1.7K
- Likes138
Check out our new paper: One Period Is Often Enough for RoPE!
Kaifeng Lyu retweeted
Wrapping up my first month at @DiscoLoopAI (am on leave from @PrincetonCS and @PrincetonPLI ). Excited to work on using AI to accelerate scientific discovery and solve important problems. My first startup experience, and with an incredible set of colleagues!
Announcing Discovery Loop!
I am very excited to announce that, along with my longtime friends and collaborators @Sanjay_Ghemawat, @OriolVinyalsML and @quocleix, we are founding Discovery Loop (@DiscoLoopAI), a Public Benefit Corporation whose mission is to automate machine learning, science, and engineering to accelerate discoveries and progress. The four of us have worked together for 14 to 30 years, and have helped build some of the world’s most used products, infrastructure and AI models, and we’re excited to turn our attention to this ambitious endeavor.
♾
Learn more at: discoveryloop.com
Kaifeng Lyu retweeted
Can LLMs generate novel research ideas? We propose Priced Guidance to quantify this via compression!
We lower bound P(LLM forecasts a future paper’s idea X). Instead of sampling ideas until X is matched, the LLM plays “20 Questions” to recover X, charging each hint by surprisal.
Kaifeng Lyu retweeted
github.com/marin-community/m…
Mini idea: reuse hidden instead of KV in YOCO seems to give a small boost for pretraining while keeping the majority of prefilling savings
Kaifeng Lyu retweeted
unfortunately i contacted them on the night of wednesday, september 2nd making the following points
1. i had intel they had information about math work i was doing and a few days before they had spun up a group to try to compete.
2. i emphasized and reemphasized mine was a strictly personal mathematical collaboration
3. that this was a collaboration that had been going on for a year
4. that in my personal collaboration we were happy users of codex as well as obviously claude (and i later complimented astra explicitly after it came out for its help, and reassured i would obviously be sprinkling acknowledgments for it, previous models, and codex liberally in the writeups)
5. that it would look terrible if openai were competing against mathematician consumers, and my collaborator (Tristan) wanted to email about this explicitly
6. that i expected none but in the worst case intended to refuse marketing around the result we were working towards, being the fruit of a personal external academic collaboration, and wanted to express this because i figured worries about such marketing by their leadership was why a team was spun up to extraordinarily directly compete.
sebastien and i have in the past months had our fun pushing each other on twitter, but as i process it all i keep feeling bad about Tristan.
1. neither of us could sleep Sunday night, so we went on a walk around the city. i finally asked him why fluids, and he told me it was the navier-stokes problem. he told me parts of the story about his classic nonuniqueness result with Vicol i hadn’t heard yet, and also about how he too grew up writing shaders as a kid and then switched to math
2. tristan is a mathematician’s mathematician. he kept rock solid to his principles during the tense calls and negotiations, immediately turning down a career dream and million dollar prize for me, the sort of strange excited unclear-quasiuseful confused idiot in our collaboration. fifteen minutes later, his feet were up on his desk with three toes per foot peeking out of ripped socks
3. our periodic call after i’d texted him euler blowup was just laughing for an hour or two. just laughing. i’d seen fable deal correctly with monge ampere in the equality case of ehrhart’s conjecture, so in some call before that i’d really pushed him for the craziest shit he could think of that i might try. i texted him the pdf as an update on that “diego luis 3” line, in which following his incredible expression of intuition i hodgepodge slapped together a bunch of our recent work, in particular correcting a bunch of better and lesser known papers in the literature, as well as on diego and luis’ ipm paper, and hoped an analogous construction for boussinesq (thus axisym euler) might work out. i managed to wade through enough confusion to insist to the model it stop telling me stupid regularity stuff about kelvin waves, and a rayleigh-taylor ansatz stuck. like i said, just laughing. when i texted him the pdf, the worst thing either of us had seen so far, i didn’t even comment on the euler blowup claim, only explaining why skepticism should remain for boussinesq, because i found it so shocking. i mean what the hell did i know, i must’ve confused myself by misunderstanding one adjective like i was used to with algebraic geometry. im so lucky to have been able to live that
4. he managed all this this entire time with unclear sleep, timestamps continue to amaze me, due to just recently having a kid
it is obvious he deserves the same sort of praise he heaped upon luis, perhaps mutatis mutandis:)
Kaifeng Lyu retweeted
Can Looped Transformers still help when FLOPs, parameters, and KV cache are all matched?
We introduce SMELT: Scaling Laws for Compute-Matched MoE Looped Transformers.
The answer is yes. And the advantage grows with scale. 🧵 1/8
Paper: arxiv.org/abs/2609.01343
Thanks for posting our work, Zhanpeng! The effective LR perspective is indeed very useful for training LLMs.
arxiv.org/pdf/2608.27370
Detailed technical report with tons of insights!
Notably, they uses MuonH as optimizers and discussed Effective LR with similar insights as in our blog hy.tencent.ai/research/elr?l….
@vfleaking is making ELR great again!
Excited to introduce Puro-2B, along with its fully open recipe that lets you train a Qwen2-1.5B-level LLM on RTX 5090 GPUs for less than $5090!
As someone who has always been curious about why LLMs work so well, one of the biggest challenges today is that we not only lack a good theory, but are also losing the ability to run controlled training experiments at scale, systematically vary key factors, and gather the empirical evidence needed to understand how LLMs learn from data.
Training LLMs has simply become too expensive.
There are now a few great open or almost open training recipes for models with billions of parameters, but reproducing them can still cost hundreds of thousands of dollars. As this makes many basic questions about LLMs inaccessible to academic labs, we really hope to bring this cost down far enough that academic labs can train and study genuinely capable models from scratch.
Puro is a full-stack engineering effort to reduce training cost, which mixes engineering and research across data, optimization, numerical precision, hardware, and training strategy. It also builds on some of our own previous research:
📈 Curriculum Model Averaging (CMA): data curriculum designs have to work together with moderate learning rate decay and model averaging (arxiv.org/abs/2511.18903)
◉ Hyperball / MuonH: an optimizer wrapper that constrains weight and update norms, and directly controls the effective (angular) step sizes for stable and efficient training (arxiv.org/abs/2606.16899)
There is still a long way to go, and we would be very happy to hear your feedback and ideas!
Kaifeng Lyu retweeted
We submit MuonH with minus-sqrt LR scheduler (1-\sqrt{t}) to the modded-nanogpt Track 3 benchmark.
It improves the current MuonH SOTA by 75 steps—and, surprisingly, nearly doubles the peak learning rate under this schedule.
More results are on the way. Stay tuned.
Many thanks for highlighting our work!
Interesting finding on frontier models.
It turns out that frontier models can write proofs but stumble on faithfully copying a long block of text.
This has huge implications.
It sounds trivial, which is exactly why it is surprising. Exact reproduction of code, tables, and structured documents is something people quietly rely on every day when using frontier LLMs.
The authors measure this copying failure and show that treating text as a 2D layout, rather than a 1D token stream, recovers much of the lost fidelity.
Paper: arxiv.org/abs/2607.16072
Learn to build effective AI agents in our academy: academy.dair.ai/
Kaifeng Lyu retweeted
Finally, our paper is on arXiv! 🎉
Huge thanks again to my amazing collaborators Yiran @zhangyir22, Yingfa @yingfachen, and Kaifeng @vfleaking! 🙌
We complete the global-minima analysis of 2D-RoPE and prove that an $\ell_\infty$-constrained global minimizer found by one-layer 2D-RoPE can learn copy with length generalization.
More broadly, we hope this sheds light on how positional encodings shape a model’s learning behavior beyond this specific setting. ✨
arXiv: arxiv.org/abs/2607.16072
P.S. Always happy to chat about theory! 🤩
(1/n)If you paste your data into an LLM and ask it to plot it in Python, it first has to copy every number into the code — so can you trust the result, or might it quietly miscopy a few?
Our new paper finds frontier LMs sometimes do get this wrong! Models that can solve hard reasoning problems still fail at a much simpler task: copying an input exactly, token by token.
Why? RoPE has an inductive bias that works against exact copying.
Better option? we propose 2D-RoPE, which views text as 2D instead of 1D.
Kaifeng Lyu retweeted
To figure out the mechanism of an unfamiliar environment, an agent has to work out what the world is and how it changes. That's what our [schema] does.
On ARC-AGI-3 Public, [schema] thinks like a physicist: it grounds the state and discovers the mechanism in one editable program. This induced world program makes action efficient: once it discovers a mechanism, it plans inside the model for free. And we achieve an amazing 99% RHAE.
Starting from here, mechanism discovery in much richer environments is what's next. Stay tuned!
Alignment should not be done at the post-training stage alone, and pretraining-stage alignment should go beyond just trying to make data safe!
Check out our new paper:
Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
Existing pretraining-stage alignment methods use data filtering and rewriting to construct a safe pretraining corpus. But is safe data enough for building a safe model? Of course not!
We introduce a pretraining-stage alignment method beyond just making data safe: Safety Reflection Pretraining (SRP). By teaching models to self-monitor their generations and halt unsafe behaviors, SRP significantly reduces attack success rates on inference-stage and finetuning attacks!
Check out our new paper: Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection arxiv.org/abs/2606.19168
🧵👇 (1/n)
Kaifeng Lyu retweeted
An important law that a lot of people don't know in reasoning composition tasks: Power law > Uniform.
**Skew** makes things learn faster!
A question on synthetic data generation: If we want a language model to solve k-step arithmetic problems (such as a+b*c-d=?), with operands from 1 to 100, which training distribution should we use?
A. Uniform distribution: Sample these k operands uniformly from 1 to 100
B. Power law: randomly shuffle 1-100 and impose an artificial power law. Sample these k operands according to this power law.
⚡Our ICML 2026 (spotlight) paper shows: Option B is better! Surprisingly, the same idea extends far beyond this simple example to many reasoning tasks that require implicit composition of multiple atomic skills, including multi-hop QAs and synthetic GSM problems.
📄Paper: arxiv.org/abs/2604.22951
📝Blog: zixuan-wang-dlt.github.io/po…
Kaifeng Lyu retweeted
Every data curation pipeline shares the same instinct: dedup, rebalance, flatten toward uniform. This work shows the opposite for compositional reasoning: uniform sampling creates a symmetric, nearly-flat loss landscape at init, and GD can't find a descent direction (they prove a CSQ lower bound!).
Zipf's law in natural language might not be a bug we need to fix. It might be part of why pretraining learns to reason at all.
A question on synthetic data generation: If we want a language model to solve k-step arithmetic problems (such as a+b*c-d=?), with operands from 1 to 100, which training distribution should we use?
A. Uniform distribution: Sample these k operands uniformly from 1 to 100
B. Power law: randomly shuffle 1-100 and impose an artificial power law. Sample these k operands according to this power law.
⚡Our ICML 2026 (spotlight) paper shows: Option B is better! Surprisingly, the same idea extends far beyond this simple example to many reasoning tasks that require implicit composition of multiple atomic skills, including multi-hop QAs and synthetic GSM problems.
📄Paper: arxiv.org/abs/2604.22951
📝Blog: zixuan-wang-dlt.github.io/po…
If you happen to care about a reasoning task that requires implicit compositions of multiple atomic skills, and you want to generate synthetic data to teach an LLM this task, should you make the skill distribution as uniform as possible to ensure coverage?
Our ICML 2026 (spotlight) paper shows the opposite: inducing an artificial power-law distribution over skills can make models learn much better, even when the test distribution is uniform!
Poster Session:
📝 The Power of Power Law: Asymmetry Enables Compositional Reasoning
📅 5:00 PM–6:45 PM, July 9, 2026
📍 Hall A, #1200
A question on synthetic data generation: If we want a language model to solve k-step arithmetic problems (such as a+b*c-d=?), with operands from 1 to 100, which training distribution should we use?
A. Uniform distribution: Sample these k operands uniformly from 1 to 100
B. Power law: randomly shuffle 1-100 and impose an artificial power law. Sample these k operands according to this power law.
⚡Our ICML 2026 (spotlight) paper shows: Option B is better! Surprisingly, the same idea extends far beyond this simple example to many reasoning tasks that require implicit composition of multiple atomic skills, including multi-hop QAs and synthetic GSM problems.
📄Paper: arxiv.org/abs/2604.22951
📝Blog: zixuan-wang-dlt.github.io/po…
In general, the best training distribution for teaching a model a task is not necessarily similar to the test distribution. Many other mechanisms can lead to this. Interested? Check out our prior work with Marko Medvedev, Zhiyuan Li, and Nati Srebro: arxiv.org/abs/2510.25108
Kaifeng Lyu retweeted
We’re entering a data-scarce regime; but is less data always worse? Our #ICML work shows that small-set training (e.g., with repetition) can be actively leveraged as a useful optimization bias for structured reasoning tasks. Come chat with Jingwen and Ezra in Seoul to know more!
Kaifeng Lyu retweeted
Excited to share our ICML 2026 paper "SPA: A Simple but Tough-to-Beat Baseline for Knowledge Injection"!
Our finding might be surprising: for knowledge injection, just 7 carefully designed prompt templates, scaled up to generate synthetic data, can outperform several complex methods across three benchmarks, and stay robust across different generator and target model families.
By showing the effectiveness of this simple baseline, we hope it serves as a useful reference point for future work on scalable, robust, and principled synthetic data generation for continual pretraining.
🔈P.S. I'm actively seeking PhD positions starting Fall 2027. Please feel free to reach out or grab a coffee at ICML!!
A bitter lesson for knowledge injection: simple prompt engineering, when scaled to generate large amounts of synthetic data, can beat more complex methods.
To inject knowledge into LLMs for data-scarce domains, prior work builds complex methods for synthetic data generation: some are based on RL to optimize the generator, and some involve multi-stage prompting pipelines.
In our ICML 2026 paper, we propose SPA (Scaling Prompt-engineered Augmentation), a simple but tough-to-beat baseline: built from just 7 prompt templates, it outperforms several strong methods. 🧵
When exploring methods for knowledge injection through synthetic data generation, we found that as the token budget scales up, these methods fail to outperform a very simple baseline: Repeatedly prompting an LLM to rewrite source documents using prompt templates sampled from a carefully curated set.
We call this method Scaling Prompt-engineered Augmentation (SPA).
A bitter lesson: at scale, simple methods can beat more complex ones.
A bitter lesson for knowledge injection: simple prompt engineering, when scaled to generate large amounts of synthetic data, can beat more complex methods.
To inject knowledge into LLMs for data-scarce domains, prior work builds complex methods for synthetic data generation: some are based on RL to optimize the generator, and some involve multi-stage prompting pipelines.
In our ICML 2026 paper, we propose SPA (Scaling Prompt-engineered Augmentation), a simple but tough-to-beat baseline: built from just 7 prompt templates, it outperforms several strong methods. 🧵