Pinned Tweet
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Can reasoning finish while latent states keep moving?
Happy to share Quotient Recurrent Reasoner (QRR)!
We let symmetry-generated rotations persist while task-relevant dynamics converge faster in quotient space, reaching 97.9% on Sudoku-Extreme with 78% fewer loops than FPRM.
David González-Martínez retweeted
MesaNet rotated 90 degree?
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
delta rule rotated by 90°
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
We’re bringing natural-language control to any scientific generator.
We are very excited to introduce LLM-to-Flow Distillation (LFD), a method to adapt scientific generators to high-level natural-language discovery goals.
Starting from textual goals like “design peptides with promising anticancer activity” or “design small molecules binding to the D2 dopamine receptor”, LFD distills high-level goal-related knowledge from LLMs into scientific generators through iterative DPO on LLM-generated sub-goals.
🔥 LFD does not need task-specific proxy rewards, works with any frontier LLM via API-access, and outperforms state-of-the-art reward-guided methods such as FlowGRPO and DiffusionNFT on held-out computational evaluations. (1/6)
David González-Martínez retweeted
mHC, iHC, oHC, xHC, and a zoo of AttnRes variants... An LLM architect with FOMO from the daily explosion of new papers? 🫠
Fear not! You can't afford to miss ScalingDepth — our community hub for keeping up with the ever-evolving world of residual connections!
We'll keep tracking and sharing the latest research, methods, and papers from the community 🚀
And it's an OPEN community! Missed something? PRs and DMs are always welcome! Got a cool blog, post, idea, or unexpected finding? We'd love to hear it! ScalingDepth hosts a dedicated space to showcase community insights and discoveries.🌍
Built by the community. Driven by curiosity. Pushing the limits of depth! 🔥
🌐 keyu-wang-2002.github.io/Sca…
David González-Martínez retweeted
Depth Memory augments residual streams with compressed, queryable memory, outperforming PreLN, mHC, and AttnRes on perplexity at 400M and 1.5B parameters. Check out the paper! 😍
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
🥇 LDM beats AttnRes and mHC on DepthBench, and it’s the method that best turns extra depth into lower loss.
David González-Martínez retweeted
Introducing Depth Memory 🧠
One framework that unifies residual methods like mHC and AttnRes. They turn out to be special cases of a single view: how layers write to and read from a memory across depth.
Our new depth mixer, Linear Depth Memory (LDM), consistently beats both at minimal overhead. 👇
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Unifying attention residuals and mHC by introducing a memory stream along depth 🤯 can be instantiated with a linear attention mechanism. And it’s fast!
Such an exciting project led by our students @_davidgonmar and @philna00
👇
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Our new preprint Depth Memory is finally live!
We propose a framework that unifies residual methods such as mHC and AttnRes and propose Linear Depth Memory, a novel depth mixer. Check it out!
Interactive blog post: phnazari.github.io/blog/2026…
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Great paper, check ti out:
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
Appreciate it! In fact, it’s even better.
Since we don’t have to worry about parallelization, we can solve the regression objective exactly.
So LDM is more like MesaLayer over depth.
In fact, our ablations show the delta rule is good. But RLS is even better! 🚀
David González-Martínez retweeted
Super interesting work applying test-time regression along the depth axis
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
David González-Martínez retweeted
🚀Also make sure to check out our blog post! phnazari.github.io/blog/2026…
David González-Martínez retweeted
super cool to see that Linear Depth Memory achieves impressive performance on DepthBench @_davidgonmar.
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
We also evaluate on DepthBench's standard settings (see @Keyuciallo's pinned tweet). At 400M, LDM favors more shallow models and beats all baselines except HC in the deepest setting. At 1.6B, it improves with depth, beating PreLN, HC and AttnRes.
Joint work with @philna00 (co-first author), @tk_rusch and @Shiwei_Liu66.
Paper: arxiv.org/abs/2610.18809
Code: github.com/davidgonmar/depth…
Checkpoints: huggingface.co/collections/d…
