Pinned Tweet
Could this be a direction for future model architectures?
Introducing Memory Attention (MA).
alphaxiv.org/abs/2609.28399
你才是真正的openai，友商又要难受了😭
🚀 Introducing DeepInfer, our in-house inference framework, now fully open source!
Powering DeepSeek-V4.1-Flash at scale with high throughput and low latency. We’re sharing our production stack to help the community lower compute costs, reduce energy use, and make AI more affordable.
🔗 github.com/deepseek-ai/DeepI…
1/4
From Astra to Opus 5.5, the past few months have arguably been the darkest days for Chinese LLMs.
But now, DSV4.1 is coming back from hell. 🐋
The pressure is back on everyone else.
👀 Rumor has it that Kimi K3.1 and MiniMax M3 Pro are both dropping by the end of this month.
🚀 DeepSeek-V4.1-Flash-1017 is here!
Stronger benchmark performance, faster task completion, and shorter outputs than the previous version—a Pareto improvement in performance and cost. Now live on our website, app, and API.
🛠️ Optimized for @DeepSeekHarness, Tencent CodeBuddy, WorkBuddy, and more. Upgrade to DeepSeek Harness v0.2.1-rc.1 or later for the best experience with the new model.
1/4
Memory is everywhere now.
Memory Attention
xitter.cf/Jo1uck/status/21029865…
Memory Agents
xitter.cf/TencentHunyuan/status/…
Memory Depth
xitter.cf/_davidgonmar/status/21…
What's next? 👀
A 3B active model beats Opus 5 at memory. We open-sourced the full stack — model, engine, and app.
Hy Memory handles chat, work, and code memory in one layer, and it's first on all four benchmarks we ran, leading MemoryArena by 10.2 points while cutting memory volume 51.8% and recall tokens 39.3%.
The model isn't tied to our engine. Drop Hy-Memory-30B-A3B into Mem0, MemOS, or EverMemOS and it still comes out on top, even on the one system it never trained on. It's already running in our own products, where one assistant's multi-hop recall went from 50% to 90%. It also unifies memory across Codex, Cursor, WorkBuddy, and more.
Repo: github.com/Tencent-Hunyuan/H…
Weights: huggingface.co/tencent/Hy-Me…
Questions welcome in the replies.
J.L retweeted
🚨Agentic Multimodal ICL: Can an MLLM learn to build its own in-context examples: what to look for, what to keep, and when it has enough?
We post-train MLLM agents to do this with tailored tools. ActICL beats similarity retrieval by +11.6 points, and boosts frontier models including GPT 5.6!
Key takeaways:
1️⃣ We cast ICL retrieval/context construction as an agentic task: the agent searches a demonstration bank with retrieval goals it writes itself, reads promising candidates, curates its context, and commits it, all under token budgets.
2️⃣ We train ActICL agents with rejection-sampling SFT followed by joint RL over acquisition and answering. Across 7 benchmarks and 3 backbones, ActICL beats the strongest trained baseline by +4.4 points while using only ~3.5 demonstrations on average.
3️⃣ ActICL contexts transfer: a 9B ActICL agent builds contexts that raise GPT-5.6-Sol by +3.5 points over standard retrieval, and it beats a Gemini3.5-Flash-powered agentic workflow on all 9 model–benchmark pairs we test.
👇🧵
Memory Attention paper update!
Added extensive ablation studies exploring MA integration, placement, normalization, and Attnres.
simple idea: V = K + M.
More parameters, yet a lower GPU memory footprint, better performance, and no V cache!
💻 joluck.github.io/memory-atte…
Could this be a direction for future model architectures?
Introducing Memory Attention (MA).
alphaxiv.org/abs/2609.28399
J.L retweeted
Dimuon is a new optimizer we built for LoRA fine-tuning. In our SFT tests, it often reaches lower loss in fewer steps than AdamW. We're still experimenting with Dimuon; you can Tinker with it today and share your feedback!
See our optimizers guide at tinker-docs.thinkingmachines…
J.L retweeted
This is Xiuyu Li. My account was stolen after 2FA was bypassed. Sorry if you got weird DMs. That was not me.
The account is now suspended and I cannot reset the password. I am still trying to get back in and will contact @X support to restore it.
Phishing attacks through @StepFun_ai verified account
please look into this @ContentCase and everyone be safe , they hacked many ai labs employees accounts to do it
My review of gemini 4 argon is taking time ( i had bad fever for last two days 😭)
J.L retweeted
very interesting!
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 arxiv.org/abs/2609.32534
J.L retweeted
We have evaluated all kinds of depth scaling methods, ranging from pre-LN scaling to Hyper-connections, mHC and attention residuals. Each shows quite different scaling behavior and computational depth. In our results, we found that Hyper-connections work surprisingly well.
Check out our full results at arxiv.org/abs/2609.32534.
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 arxiv.org/abs/2609.32534
Our new paper hopes this can contribute to future architecture design.
🔥 Frontier labs are redesigning the residual stream to beat the curse of depth: Kimi K3 uses AttnRes, DeepSeek V4 uses mHC, ByteDance proposed HC, and there are more: LNS, KEEL, MoDA…
🙋 But each method was tested with its own training budget, model design and codebase, so the results can't be compared directly. Which one actually works best? More importantly, which one turns architectural depth into effective computational depth?
👇 We built DepthBench to find out.
📄 arxiv.org/abs/2609.32534
J.L retweeted
"Memory Attention"
Instead of computing a completely new value for every token, this paper reuses the contextual key and adds a learned memory of what that token generally represents.
So rather than generating everything dynamically with dense compute, part of the model’s knowledge becomes a cheap lookup that can even live on CPU, adding capacity without proportionally increasing GPU compute.
alphaxiv.org/abs/2609.28399
Some preliminary ablation results suggest that reusing keys in V=K+M is an effective architectural choice. We still need to validate this at larger scales, but hope these findings are useful!
Could this be a direction for future model architectures?
Introducing Memory Attention (MA).
alphaxiv.org/abs/2609.28399
感谢您的认可，Memory-Attention 仍有许多秘密需要探索，我也在补充大量的ablation实验以及大家最关心的scaling up
中秋快乐！刚刚看到一篇新论文 Memory Attention，感觉可能是个划时代的新注意力架构，给大家分享一下。
MA这篇的核心是Transformer 里的 V，可能根本没必要每次都重新算。
所以直接开始动 Attention 里最经典的 QKV 结构了。
传统 Attention 里，每个 token 都要从当前 hidden state 重新计算：
Q = XWq
K = XWk
V = XWv
但本文作者 @Jo1uck 问了一个很朴素的问题：
V 里面有多少东西，真的需要每次根据上下文重新算？
同一个 token 在不同句子里，显然存在大量可以复用的信息。
于是他们参考 Engram 这类 Memory 架构，给每个 token 建了一套可学习的 memory，然后更激进地把独立的 value projection 直接干掉：
V = K + Memory[token]
K 负责保留上下文信息，Memory 负责保存 token 自身可以复用的信息。
这样推理时，原来计算 V 的一次矩阵乘法，基本就变成了：查表 + 加法（这会大幅减少计算量）。
然后事情开始变得有意思了。
因为 Memory 只跟 token ID 和层数有关，所以甚至可以提前知道要读哪块数据，直接把这部分参数扔到 CPU，GPU 需要的时候提前 Prefetch。
实验里，MA-Offload 模型总参数是普通模型的 2.08 倍，但 GPU 上实际常驻的参数反而比普通 Transformer 少了 7.38%，推理延迟基本维持在同一水平。
训练上，在两个不同规模实验里，达到相同 loss 所需 token 数分别提升到了 1.42× 和 1.16× 的 token efficiency，也就是大约少用 29.6% 和 13.8% 的训练 token。
下游平均成绩也基本都比对应的 Standard Attention 更高。
更骚的是，作者还提出了一个 MA-Recall：
既然历史 V 可以通过 K + token memory 重新构造，那么理论上甚至不用一直保存完整的 V Cache，只保存 K 和 token ID，在需要的时候把 V 重建出来。
按论文的理论计算，这部分 KV Cache 可以砍掉接近一半，不过目前还只是分析，没有进入实际推理实验。
我觉得这篇的方向挺值得关注的。
过去大家做 Memory，更多是在 Transformer 外面继续加东西。
Memory Attention 开始反过来问：既然 Memory 已经能记住这些东西，那 Transformer 里面原来那些计算，是不是可以直接删掉？
当然论文自己写的也很克制：MA 用了更多参数，所以目前的收益还不能证明全部来自架构本身，而不是单纯的参数量增加。
但如果这条路继续成立，下一代模型的变化可能不只是 Attention 越做越复杂。
也可能是我们用了快十年的 QKV，终于开始有人认真考虑：哪些东西其实根本不用算。
MA is related to this work by @HeMuyu0327. We will cite it in the next revision.
xitter.cf/HeMuyu0327/status/2060…
In our new paper, we naturally derive a new attention variant based on the surprising finding that deep layers benefit the most from learning a context-free value vectors, without the input from the residual stream.
The attention variant: since the value vector does not depend on the context, it can be directly learned as sparse model parameter, and stored in a table of value vectors for the current layer.
The idea of having a value vector table is not new. Nanochat, for example, has it. But it was always learned as an extra group of weights to add to the existing value vector. With the insight that the deep layer **only** need a context-free vector, we can rewrite attention completely, by making the table the only source of value vectors.
On two model sizes we see that it significantly outperforms the standard attention baseline on both validation loss and benchmark scores, and slightly surpasses the strictly more expressive nanochat variant.
Besides the boost in performance, some very interesting consequences emerge in terms of memory:
- We do not need V cache anymore for the deep layers, and at long context it saves memory.
- Because we only need the token IDs for fetching value vectors, we can offload the table entirely, and just prefetch the relevant entries when previous layers are busy computing.
Me and @YuchenL52766559 are fascinated by this attention architecture's unique properties, and are doing experiments on bigger scales with it.
Paper: github.com/RiddleHe/nanochat…
MA is also related to DeepEmbedAttention(dea. We will add a comparison in v2.
Source of dea: xitter.cf/BlinkDL_AI/status/1939…
