The gains extend beyond knowledge recall.
Compared with continued dense training under the same token budget, STAPLE improves OLMo-2 1B by:
• +4.27 points on HumanEval
• +2.80 on MBPP
• +2.27 on GSM8K
• +1.65 on DROP
STAPLE achieves a higher ten-task average than the static sparse alternatives such as STEM, Engram-style, as well as a serving-oriented MoE upcycling baseline with eight experts and top-1 routing. We use 100B tokens for warmup and another 100B for midtraining.
The benefits extend to OLMo-2 7B despite shorter training budget (30B warmup + 70B midtraining): average math performance improves by 4.2% and code by 3.7% relative to dense continuation, with improvements on 9 of 10 tasks. MMLU gains are smaller, potentially reflecting the shorter warmup.
Billions of added parameters do not have to mean billions of extra parameters resident on GPU.
STAPLE-12L keeps 19.73 GB of embedding tables on CPU, using a 1 GB GPU cache. Predictable access enables prefetching; sparse training updates touch only the rows used by the batch.
The result:
• Decoding throughput within 1.5% of dense
• Training throughput within 3% of dense
• About 2× the decoding throughput of our MoE baseline
• 58% lower GPU energy per token than our MoE baseline, with only 1.2% overhead over dense
What does the new memory actually contribute?
Removing the learned embeddings sharply degrades math and code performance, showing that the model has learned to rely on them.
The memory is interpretable and editable, while remaining responsive to context - a promising foundation for expanding pretrained models with additional complementary knowledge.
🧠 How can we keep scaling LLMs with minimal changes in their serving energy?
DeepSeek’s Engram and our concurrent work STEM showed the promise of static sparse memory: large embedding tables with efficient access. But can mature pretrained models learn to use a newly added memory within a limited training budget, avoiding costly pretraining from scratch?
🚀 Introducing STAPLE: a recipe to extend pretrained models' abilities with static sparse memory. Serving energy per token stays close to dense and >50% below MoE upcycling in high-throughput regime.
Scalable capacity, interpretable memory, and compatibility with MoE upcycling - without any meaningful increase in serving cost. 🧵
Paper: arxiv.org/abs/2610.12599
Code: github.com/Infini-AI-Lab/STA…
Website: infini-ai-lab.github.io/STAP…
Our central insight: under a limited training budget, new memory should complement pretrained knowledge.
STEM replaces FFN up-projections with token-indexed embeddings. For upcycling, STAPLE retains those projections and adds the embeddings alongside them, under the same context-dependent gate.
The recipe:
• Zero-initialize the embeddings to preserve the starting model’s function.
• Warm up with the retained up-projections frozen.
• Then jointly mid-train the model and memory.
Retaining the pretrained pathway makes a substantial difference. Warmup further improves knowledge and commonsense performance.
The architecture and training schedule both matter.
Why static sparsity for expanding pretrained models?
MoE upcycling can add capacity without restarting pretraining. But cloned experts can converge to homogeneous experts, might need additional tricks and more training budget to differentiate, while larger expert pools increase GPU-memory demands and can slow serving.
Static memory offers input-specific lookups with no expert homogeneity issues. Additionally, they can keep large tables on CPU and efficiently retrieve only the needed rows.
The challenge is making that memory useful quickly. Token frequencies are highly uneven, leaving rare rows with few updates. Replacing pretrained computation also discards useful features that must be relearned.
How do we add capacity while preserving what the model already knows?
Infini-AI-Lab retweeted
Serving data is an underestimated gold mine for agent self-improvement. But how well can agents actually learn from their serving experience?
In our recent work, ServeLearnBench, we take a closer look at this question. We find that current agents still struggle to learn from experience, and effective exploration plays a surprisingly important role.
Check out more results in our paper and repo!
📄 Paper: arxiv.org/abs/2610.07792
💻 Code & Dataset: github.com/Infini-AI-Lab/Ser…
🌐 Website: infini-ai-lab.github.io/Serv…
How well can agents continually self-improve from serving experience? We’re excited to introduce our new benchmark, 𝐒𝐞𝐫𝐯𝐞𝐋𝐞𝐚𝐫𝐧𝐁𝐞𝐧𝐜𝐡:
Real-world deployment is not a one-time test of capability. The knowledge agents need is often hidden and can change over time.
ServeLearnBench tests whether agents can infer, apply, and revise this knowledge from interaction.
Across 5 learning harnesses × 6 models, we find:
1. 𝐋𝐚𝐫𝐠𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐠𝐚𝐩𝐬 𝐫𝐞𝐦𝐚𝐢𝐧. Agents can solve tasks when the hidden policy is given, but struggle to discover it from experience.
2. 𝐍𝐨 𝐟𝐫𝐞𝐞 𝐥𝐮𝐧𝐜𝐡 𝐟𝐨𝐫 𝐚𝐝𝐚𝐩𝐭𝐚𝐭𝐢𝐨𝐧. Continual learning can be costly—and can even degrade already-correct behavior.
3. 𝐄𝐱𝐩𝐥𝐨𝐫𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐚 𝐤𝐞𝐲 𝐛𝐨𝐭𝐭𝐥𝐞𝐧𝐞𝐜𝐤. Better exploration strongly correlates with better adaptation.
Infini-AI-Lab retweeted
The next frontier would be agents that get smarter as you use them. 🚀 Our new ServeLearnBench puts continual self-improvement to the test—and reveals what’s holding it back.
How well can agents continually self-improve from serving experience? We’re excited to introduce our new benchmark, 𝐒𝐞𝐫𝐯𝐞𝐋𝐞𝐚𝐫𝐧𝐁𝐞𝐧𝐜𝐡:
Real-world deployment is not a one-time test of capability. The knowledge agents need is often hidden and can change over time.
ServeLearnBench tests whether agents can infer, apply, and revise this knowledge from interaction.
Across 5 learning harnesses × 6 models, we find:
1. 𝐋𝐚𝐫𝐠𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐠𝐚𝐩𝐬 𝐫𝐞𝐦𝐚𝐢𝐧. Agents can solve tasks when the hidden policy is given, but struggle to discover it from experience.
2. 𝐍𝐨 𝐟𝐫𝐞𝐞 𝐥𝐮𝐧𝐜𝐡 𝐟𝐨𝐫 𝐚𝐝𝐚𝐩𝐭𝐚𝐭𝐢𝐨𝐧. Continual learning can be costly—and can even degrade already-correct behavior.
3. 𝐄𝐱𝐩𝐥𝐨𝐫𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐚 𝐤𝐞𝐲 𝐛𝐨𝐭𝐭𝐥𝐞𝐧𝐞𝐜𝐤. Better exploration strongly correlates with better adaptation.
How well can agents continually self-improve from serving experience? We’re excited to introduce our new benchmark, 𝐒𝐞𝐫𝐯𝐞𝐋𝐞𝐚𝐫𝐧𝐁𝐞𝐧𝐜𝐡:
Real-world deployment is not a one-time test of capability. The knowledge agents need is often hidden and can change over time.
ServeLearnBench tests whether agents can infer, apply, and revise this knowledge from interaction.
Across 5 learning harnesses × 6 models, we find:
1. 𝐋𝐚𝐫𝐠𝐞 𝐥𝐞𝐚𝐫𝐧𝐢𝐧𝐠 𝐠𝐚𝐩𝐬 𝐫𝐞𝐦𝐚𝐢𝐧. Agents can solve tasks when the hidden policy is given, but struggle to discover it from experience.
2. 𝐍𝐨 𝐟𝐫𝐞𝐞 𝐥𝐮𝐧𝐜𝐡 𝐟𝐨𝐫 𝐚𝐝𝐚𝐩𝐭𝐚𝐭𝐢𝐨𝐧. Continual learning can be costly—and can even degrade already-correct behavior.
3. 𝐄𝐱𝐩𝐥𝐨𝐫𝐚𝐭𝐢𝐨𝐧 𝐢𝐬 𝐚 𝐤𝐞𝐲 𝐛𝐨𝐭𝐭𝐥𝐞𝐧𝐞𝐜𝐤. Better exploration strongly correlates with better adaptation.
Yet a substantial learning gap remains.
Even when models have the capability to solve the task, they struggle to learn the required behavior from serving experience.
On Retail L3, the best learner reaches only 59.6%. And adaptation isn’t free: learning harnesses cost 4–101× as much as Oracle and can even hurt already-correct behavior. 💸
🧵(4/5)
Explore ServeLearnBench:
𝐖𝐞𝐛𝐬𝐢𝐭𝐞: infini-ai-lab.github.io/Serv…
𝐂𝐨𝐝𝐞&𝐃𝐚𝐭𝐚𝐬𝐞𝐭: github.com/Infini-AI-Lab/Ser…
𝐏𝐚𝐩𝐞𝐫: arxiv.org/abs/2610.07792
🧵(5/5)
We test the agent on 5 real-time apps, each deployed from a reference alone: face-to-face conversational avatar, a multimodal LLM (Qwen3-Omni) in text-prompt/speech-response mode, a video background editor, a video world model, and a video narrator.
Using Claude Code with Opus 4.8, FlashRT produces efficient real-time deployments for each one, reaching up to ~70× reduction in response latency. Its throughput gains are largest on AMD MI355X (up to 3.6× frame rate, versus 2.8× on B200), showing that FlashRT enables more scalability on platforms there's less mature expert optimization.
🧵5/5
Joint work with @wonderingkrish, @chenzhuoming911, Yanyuan Qin, Zhenyu Gu, Atri Rudra, and @BeidiChen. We are grateful for the collaboration with @AMD!
A coding agent is a good fit for this: it can read an arbitrary spec, reason about the pipeline's structure, and write a deployment tailored to it, which rule-based systems can't do.
But handing the agent a reference and asking it to make the code faster is not enough on its own. When we try prompting an agent naively, it finds some of the useful optimizations but misses others, and it rarely combines the strategies that only pay off when applied together.
🧵3/5
FlashRT guides the agent through a novel chain-of-program workflow instead:
1️⃣ Lower the reference implementation into a hierarchical graph IR, and validate that IR against the reference with a sequential interpreter.
2️⃣ Run static analysis on the IR to enumerate candidate transformations: streaming, disaggregation, model parallelism, and placement.
3️⃣ Run a measurement-gated loop: implement each variant, check its correctness against the reference, benchmark it, then re-hypothesize. Strategies that hold up get composed and ones that don't are dropped.
🧵4/5
AI is now building the infrastructure it runs on.
Getting real-time multimodal applications — including live avatars, voice agents, video world models — to actually run in real time takes serious manual systems work. Turns out a coding agent can do all the engineering work for you: up to 70× lower latency and 3.6× higher throughput, across 5 real-time apps.
Today we release FlashRT ⚡, an agent harness for real-time systems implementations and optimization. The developer writes only a simple reference spec, and the agent handles the rest end-to-end, producing efficient deployments across a range of GPU budgets in one shot, with no human intervention.
✅ Validated on 5 real-time applications that combine interactive video generation, text-to-speech, LLMs, etc.
✅ Runs on both NVIDIA and AMD GPUs; the agent optimizes for the hardware you’re on
✅ Can easily compose with third-party serving engines, the agent automatically implements orchestration
📄 Paper: arxiv.org/abs/2607.18171
🌐 Website: infini-ai-lab.github.io/flas…
🔗 GitHub: github.com/Infini-AI-Lab/Fla…
🧵1/5
Real-time multimodal applications are hard to serve because each one can compose several models (LLM, TTS, ASR, video generation DiT/VAE, etc.) into a single pipeline, and the most effective way to deploy that pipeline is different every time.
Existing tools are too rigid to keep up. Multimodal serving frameworks like vLLM-Omni apply fixed deployment policies, and auto-parallelism compilers specialize to a single workload. So for each new application, the systems work still falls to human developers.
🧵2/5
