founding mts at @PrincipiaLabs | phd @nyuniversity previously @NVIDIA, @microsoft, @princeton_nlp, @roblox
SF
Joined August 2019
- Tweets196
- Following735
- Followers1.2K
- Likes3.5K
Michael Hu retweeted
Language models learn style and content jointly.
In new preprint with @ekoermann and @kchonyc, we show that style can be discovered without supervision and controlled.
arxiv.org/abs/2610.00724
1/
Michael Hu retweeted
Can an LM, starting from random init (!!), learn to generate all of its pretraining data?
Introducing Self-Play Pretraining with Zero Data. Two models start from random initialization: a generator proposes programs for a universal Turing machine and a learner trains on their outputs. We never train on any real data, but see predictable scaling on natural datasets: zero-shot val loss on images, text, audio, and melodies decreases predictably with self-play compute. And the learner develops in-context learning capabilities.
A fun proof-of-concept, co-led with @AdityaCowsik and @KfirDolev and co-authors @gbruno_dl, @ANourya @noahdgoodman, and @YoavLevine.
OP-Mix, neural neural scaling laws, and simulating human memory are all going to NeurIPS!
see you in sydney 🇦🇺 (? provided that i can successfully turn money into a conference registration)
Michael Hu retweeted
New paper of interest to mechanistic interpretability fans!
We show that a variety of interpretability methods can be unified under a single representational hypothesis (using the Tensor Product Representation formalism).
Really exciting work by @EnyanZhang!
🚨🧵 New paper!
Mech interp has developed many ways to probe and intervene on model internals. It's natural to ask: is there a unifying picture?
In this work, we derive 4 popular interp methods from a single representational hypothesis: Tensor Product Representations (TPRs)
1/n
ALT Overview slide of "A Unifying Perspective on Language Model Representations: From Filler-Role Structure to Mechanistic Interpretability". The title is on the top, followed by authors: Zhang Enyan and Tom McCoy Left: An elephant viewed from different perspectives, where each viewer think the elephant feels like a fan, snake, wall, and rope, respectively. This classic story is compared with a black-box LLM, where it is viewed as probes, SAEs, causal traces, and analogies. Center: explanation text "We unify a variety of interpretability methods under Tensor Product Representations (TPRs), and show that they can be constructed directly from TPRs with comparable performance ", followed by a schematic of how a Tensor Product Representation (TPR) is constructed. The schematic has arrows linking to four cells, each containing an interpretability method, demonstrating they can be viewed as applications of TPR operations
Michael Hu retweeted
Finding a good training-data mixture usually means training thousands of small proxy models to search for one. Michael Hu's team got it down to one run per dataset: train a LoRA adapter on each dataset, then merge the adapters, and the merge approximates what training on that mixture would do.
This week's Summer of Data talk is live. Thanks to Michael (@michahu8, NYU) for coming by. Full talk in the reply.
Michael Hu retweeted
We discovered a third pretraining axis beyond parameters and data: exploration.
Scaling exploration monotonically improves existing models across images/video/language, and unlocks end-to-end generation.
In the simplest case, it's just a for loop.
Introducing Explorative Modeling.
TLDR:
- Gains from exploration grow with scale: 7%→36% as data scales, 13%→23% as parameters scale, and gains double at 3× the compute
- Adding exploration to ~SOTA baselines improves data efficiency by 6.2×, FLOP efficiency by 4.1×, parameter efficiency by 47%, and hits a near-SOTA 1.43 unguided FID on ImageNet
- Exploration lets you trade training compute for generalization, and scales how end-to-end your generative model is
- End-to-end Explorative Models (XMs) match diffusion performance on control tasks with up to 256× less inference compute
🧵Thread:
Heading to #ICML2026🇰🇷today! Top of mind for me:
- self-improvement & continual learning (what I'm doing this summer)
- user simulation
- data, as always (data mixing, synthetic pretraining, etc.)
Excited to meet new people! DM if you'd like to chat🥳
I'm presenting NeuNeu at the High-dimensional Learning Dynamics (HiLD) workshop next Friday, July 10:
xitter.cf/michahu8/status/201650…
super excited about using cognitive science to build better user simulators
New paper! LLM memory keeps improving, but this makes them *worse* as user sims. If we want to build models that can, e.g., simulate realistic students to train chatbots to be better teachers, then these models need to be able to forget like humans do
📄: arxiv.org/abs/2605.25680
ALT Title page of our paper, "Simulating Human Memory with Language Models"
Michael Hu retweeted
arxiv.org/abs/2605.15220
Using LoRAs for determining dataset mixture. For a continual training setup, when new datasets are introduced, it is possible to train LoRAs for them and combine them with a LoRA on previous datasets.
What is the right data mix, and how do we find it as the data keeps changing?
This is a core, unsolved problem in continual learning. To tackle it, we built a data mixing algo that works everywhere — pretraining, midtraining, instruction tuning
Introducing: On-Policy Mix
🧵1/6
And in continual instruction tuning (skill learning), OP-Mix matches the performance of self-distillation finetuning, with 95% less compute! The two methods also stack for even better performance.
5/6
in sum, you should just use OP-Mix for all of your data mixing needs, all the time 😉
📜: arxiv.org/abs/2605.15220
💻: github.com/michahu/on-policy…
6/6
