Cambridge, MA
Joined July 2023
- Tweets33
- Following286
- Followers142
- Likes96
Pinned Tweet
Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.
But what if Gated DeltaNet = Mamba-2? 👀
Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!
We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈
📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai
💻 Code: github.com/ntumm120/precondi…
Super interesting work applying test-time regression along the depth axis
mHC and AttnRes demonstrate that richer depth mixing improves model performance at scale. Is there a unifying perspective? 🤔
Meet our Depth Memory framework: each layer reads from a set of streams to form its input, then writes back its output.
Linear Depth Memory (LDM) equips models with a matrix-valued memory.
Neehal Tumma retweeted
Linear attention is arguably the most naive RNN, but still massively outperforms traditional RNNs by maintaining a matrix state. So... what if we use a (triadic) outer product of three vectors and maintain a three-dimensional state?
Introducing: Triadic Linear Attention 🧵
Neehal Tumma retweeted
SFT is not dead! 🥳
We found a way to make SFT rival current prevailing posttraining methods, often generalizing better and forgetting less than RL and OPSD. 🤯
Following our prior work on reasoning with sampling, we now introduce sampling to the posttraining stack.
1/n
PGDN and PKDA have been merged into FLA, check them out! github.com/fla-org/flash-lin…
Some say Gated DeltaNet > Mamba-2. Others say Mamba-2 > Gated DeltaNet.
But what if Gated DeltaNet = Mamba-2? 👀
Well maybe not exactly — but with least-squares preconditioning, we show that they reduce to the same recurrence!
We use this lens to design PDN, PGDN, and PKDA: preconditioned delta-style recurrences that outperform their unpreconditioned counterparts at scale 📈
📄 Paper: arxiv.org/abs/2604.21100 w/ @loo_noel @liquidai
💻 Code: github.com/ntumm120/precondi…
Neehal Tumma retweeted
Looped models scale compute by iterating in depth, demonstrating great potential in reasoning. But when should the looping stop, and how to keep it trainable at tens of thousands of unrolled layers? FPRM solves both. (1/7)
Cool paper showing the utility of data-dependent short convolutions.
One comparison I’d love to see: explicit filter conditioning used here vs the LFM style approach from @liquidai, where the short conv kernel is static but the local operator remains input-dependent via gates.
Basically dynamic vs gated static FIR filters.
Neehal Tumma retweeted
Excited to share SwitchCraft, our first step toward designing complex protein function through state switching (ligand responsive binders, allosteric control, biosensors, and more).
Stay tuned for wet-lab results from our amazing collaborators very soon!
Binder design has come of age thanks to generative models—but how can we access the wider array of dynamic, multistate protein functions, so elegantly employed by nature?
@mihirbafna14 and I are excited to share SwitchCraft, a framework for designing such functions. (1/7)
(13/N) But we think this only scratches the surface.
PDN/PGDN/PKDA are just three points in a larger design space: different recurrence families + different preconditioners.
There is a huge optimization literature on preconditioning that can be translated into recurrence design.
The bigger message: preconditioning is a useful new lever for designing linear recurrences.