PhD Candidate at Max Planck ETH Center for Learning Systems
Joined August 2015
- Tweets52
- Following359
- Followers148
- Likes17.5K
Pinned Tweet
Our new preprint Depth Memory is finally live!
We propose a framework that unifies residual methods such as mHC and AttnRes and propose Linear Depth Memory, a novel depth mixer. Check it out!
Interactive blog post: phnazari.github.io/blog/2026…
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Philipp Nazari retweeted
Very happy to share the first paper of my PhD: Normalizing Flow Maps! (NFM)⚡️
We make likelihood evaluation in flow maps fast and tractable by replacing the DiT backbone with a normalizing flow🔄.
A big thanks to @tim_hitge, @mmbronstein, @bose_joey and the rest of the team!
🚨 New Paper Alert, 🎤 Accepted for an oral presentation at the NeurIPS 2026 STODY workshop!
📣 Introducing: Normalizing Flow Maps (NFM): Combining few-step transport from Flow maps with tractable likelihoods from Normalizing Flows.
This is my first work with my first crop of new PhD students, @francescoshox and @tim_hitge, who genuinely knocked this project out of the park.
📄 Paper: arxiv.org/abs/2610.22711
📝 Blog: the-bose-lab.github.io/norma…
💻 Code: github.com/francescoshox/Nor…
🧵
1/8
Philipp Nazari retweeted
🚨 New Paper Alert, 🎤 Accepted for an oral presentation at the NeurIPS 2026 STODY workshop!
📣 Introducing: Normalizing Flow Maps (NFM): Combining few-step transport from Flow maps with tractable likelihoods from Normalizing Flows.
This is my first work with my first crop of new PhD students, @francescoshox and @tim_hitge, who genuinely knocked this project out of the park.
📄 Paper: arxiv.org/abs/2610.22711
📝 Blog: the-bose-lab.github.io/norma…
💻 Code: github.com/francescoshox/Nor…
🧵
1/8
Philipp Nazari retweeted
Our new preprint LLM-to-Flow Distillation (LFD) is finally live! 🧬
I’m convinced language is the right interface for controlling generative models. And that should apply to molecules just as much as images!
LLMs know a lot about science. DPO gives us a way to transfer that knowledge into scientific generators. With LFD, “design molecules that bind to the D2 dopamine receptor” becomes a goal we can actually train a flow model to pursue.
The LLM builds an adaptive curriculum around what the generator can currently produce. As the model improves, the teaching and guidance from the LLM change too.
The cool part is that we can do this with models we already have, without redesigning their architectures or engineering a task-specific proxy reward.
Really happy to share this work with Riccardo, Federico and the rest of the team!! Full thread below 👇
We’re bringing natural-language control to any scientific generator.
We are very excited to introduce LLM-to-Flow Distillation (LFD), a method to adapt scientific generators to high-level natural-language discovery goals.
Starting from textual goals like “design peptides with promising anticancer activity” or “design small molecules binding to the D2 dopamine receptor”, LFD distills high-level goal-related knowledge from LLMs into scientific generators through iterative DPO on LLM-generated sub-goals.
🔥 LFD does not need task-specific proxy rewards, works with any frontier LLM via API-access, and outperforms state-of-the-art reward-guided methods such as FlowGRPO and DiffusionNFT on held-out computational evaluations. (1/6)
Cool website comparing different residual methods for transformers. Check it out!
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
Philipp Nazari retweeted
Depth Memory augments residual streams with compressed, queryable memory, outperforming PreLN, mHC, and AttnRes on perplexity at 400M and 1.5B parameters. Check out the paper! 😍
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Philipp Nazari retweeted
🚀 Science is the next big challenge for agentic AI systems.
While LLMs are becoming incredibly capable, it is still an open question how we can translate their intelligence into scientific discoveries, where closing the loop is costly and reliable reward functions are often unavailable.
We introduce LLM-to-Flow Distillation (LFD): a method that combines the scientific knowledge of frontier LLMs with the structural priors of pretrained generators, allowing us to steer these latter toward natural-language discovery goals.
Check out our work in the thread 👇🏼
We’re bringing natural-language control to any scientific generator.
We are very excited to introduce LLM-to-Flow Distillation (LFD), a method to adapt scientific generators to high-level natural-language discovery goals.
Starting from textual goals like “design peptides with promising anticancer activity” or “design small molecules binding to the D2 dopamine receptor”, LFD distills high-level goal-related knowledge from LLMs into scientific generators through iterative DPO on LLM-generated sub-goals.
🔥 LFD does not need task-specific proxy rewards, works with any frontier LLM via API-access, and outperforms state-of-the-art reward-guided methods such as FlowGRPO and DiffusionNFT on held-out computational evaluations. (1/6)
Philipp Nazari retweeted
Introducing Depth Memory 🧠
One framework that unifies residual methods like mHC and AttnRes. They turn out to be special cases of a single view: how layers write to and read from a memory across depth.
Our new depth mixer, Linear Depth Memory (LDM), consistently beats both at minimal overhead. 👇
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Philipp Nazari retweeted
Very interesting work!
Seems closely related to Deep Delta Learning.
arxiv.org/abs/2601.00417
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Philipp Nazari retweeted
MesaNet rotated 90 degree?
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Philipp Nazari retweeted
Great paper, check ti out:
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Philipp Nazari retweeted
We’re bringing natural-language control to any scientific generator.
We are very excited to introduce LLM-to-Flow Distillation (LFD), a method to adapt scientific generators to high-level natural-language discovery goals.
Starting from textual goals like “design peptides with promising anticancer activity” or “design small molecules binding to the D2 dopamine receptor”, LFD distills high-level goal-related knowledge from LLMs into scientific generators through iterative DPO on LLM-generated sub-goals.
🔥 LFD does not need task-specific proxy rewards, works with any frontier LLM via API-access, and outperforms state-of-the-art reward-guided methods such as FlowGRPO and DiffusionNFT on held-out computational evaluations. (1/6)
Appreciate it! In fact, it’s even better.
Since we don’t have to worry about parallelization, we can solve the regression objective exactly.
So LDM is more like MesaLayer over depth.
In fact, our ablations show the delta rule is good. But RLS is even better! 🚀
Depth regression doesn’t profit from parallelizability. LDM thus scales the exact RLS solve.
The sequence axis requires parallelization, motivating MesaNet over MesaLayer (@oswaldjoh @osieberling).
Viable depth recurrences are a superset of viable recurrences over time 🚀
Philipp Nazari retweeted
Significant work. And I think the memory stream may have potential on controlling the recurrent depth of the model.
Philipp Nazari retweeted
super cool to see that Linear Depth Memory achieves impressive performance on DepthBench @_davidgonmar.
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Wanna bet on your paper being accepted to #ICLR? Check out acceptodds.com by the amazing @uptosomestuff, a prediction market for #ICLR submissions. Could this be a way to navigate the ever growing number of ml papers?
Philipp Nazari retweeted
Amazing work!! Very proud to be part of the team🙂
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 arxiv.org/abs/2609.32534
Philipp Nazari retweeted
PGDN and PKDA have been merged into FLA, check them out! github.com/fla-org/flash-lin…
Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.
But what if Gated DeltaNet = Mamba-2? 👀
Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!
We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈
📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai
💻 Code: github.com/ntumm120/precondi…
Philipp Nazari retweeted
It seems like the Ellis Institute Tübingen is the place to be in Germany when you want to work on LLMs
