Leading LLM Efficiency Research @ Together AI. prev: @awscloud @MSFTResearch, @Cornell PhD.
Earth, the Milky Way.
Joined July 2014
- Tweets412
- Following776
- Followers1.1K
- Likes2.6K
Thanks for the shout out! This work takes quantization to the next level by avoiding straight through gradient estimator for QAT and preventing quality drop. Glad to see it being adopted!
arxiv.org/pdf/2608.13966
Tested the QUASAR-QAT Qwen3.8-27B NVFP4 build on a single DGX Spark last night. vLLM, util 0.37, 131k context.
Plain decode: 12.3 tok/s
With MTP speculation (2 draft tokens): 21.8 tok/s, 82% of drafts accepted
vLLM 0.28 vs 0.26: no difference on either setting
Structured output and tool calls: 9 of 9 valid on every run
95k-token prompt answered in 9 seconds
Likelihood 0.5156 bits/char with speculation off. Within 0.8% of the unquantised BF16 base and the best NVFP4 27B build I have measured
43 GB resident, so there is room for other work on the box
Checkpoint: QUASAR-QAT/Qwen3.8-27B-QUASAR-NVFP4 on Hugging Face. Quantisation-aware training by @counathe, @ben_athi, @chrismdesa and Tianyi Zhang (@togethercompute and Cornell), arXiv 2608.13966.
Full board, with the other 27B builds beside it, on the single-Spark page:
scottleimroth.com/ai-technol…
Ben Athiwaratkun retweeted
How does one RL post-train a 397B model for long-horizon knowledge work? 👩💼
We share every step we took to bring Qwen 3.5 397B from 16.1% Pass@1 to 27.3% on APEX-Agents using DPPO, including final models weights and the full training script🚀 This is the first of many works from Mercor Research on open model training research.
Full blog: mercor.com/blog/training-fro…
Source code: github.com/Mercor-Intelligen…
Ben Athiwaratkun retweeted
Read the blog: together.ai/blog/icml-2026
See us at ICML: together.ai/icml-2026
Ben Athiwaratkun retweeted
A year ago this would have been an obvious closed-model task.
Now GLM-5.2 can read the issue, reason through the scene, patch the code, and keep moving on Together AI.
@togethercompute + @Zai_org GLM 5.2 are a dope combination. The inference is just so fast... I'm no pro with three js, and this is probably amateur work... But Opus 4.8 was having trouble with the 3d -> 2d transition. I started debuggin with GLM 5.2 ... #win
Ben Athiwaratkun retweeted
Exciting launch for OSS models & @togethercompute - check it out!! 👇
Closed-source models aren't worth the premium.
We generated 12 landing pages with Kimi K2.7 Code and Claude Fable 5. Kimi came in 16x cheaper with comparable quality, especially once we gave it visual context through a design MCP server.
Open-source models are already a practical choice for this kind of workflow.
Ben Athiwaratkun retweeted
GLM models continue their amazing trajectory with GLM-5.2 that supports 1M context window.
We continue to see amazing innovation from open-weights models on long-context. GLM-5.2 extends DeepSeek's sparse attention by reusing the same indexer across every four sparse attention layers, reducing per-token flops by ~3x at 1M context.
Introducing GLM-5.2: Frontier Intelligence, Open Weights
- Significant improvements in coding and agentic tasks
- Strong long-horizon capabilities with a 1M context window
- Two levels of reasoning effort: GLM-5.2 (max) pushes the limits, while GLM-5.2 (high) strikes a strong balance between performance and token efficiency
- MIT-licensed open weights
- Same API pricing as GLM-5.1
Tech Blog: z.ai/blog/glm-5.2
Weights: huggingface.co/zai-org/GLM-5…
API: docs.z.ai/guides/llm/glm-5.2
Coding Plan: z.ai/subscribe
Chat: chat.z.ai
Ben Athiwaratkun retweeted
.@DecagonAI cut voice agent cost per turn nearly 6x with Together AI.
They moved from closed models to fine-tuned open models, while keeping latency low enough for real-time voice:
→ <400ms p95 model latency per turn
→ custom speculators and prompt caching
→ optimized serving on NVIDIA Blackwell
→ weekly, sometimes daily model deployment velocity
This is the closed-to-open shift: more control, better tokenomics, and production performance without being locked into proprietary APIs.
Ben Athiwaratkun retweeted
Optimizing GLM 5.1 came down to three things:
-> Rewrote the indexer topk kernel
-> Fused the indexer kernel to reduce memory and launch overhead
-> Eliminated CPU overhead that was gating prefill throughput
The bigger win was in the indexer. Once we fixed that, the rest made it even faster.
GLM 5.1 is available on Together AI.
Great to see ThunderAgent research from @togethercompute (KV aware routing for agentic inference) being adopted by @NVIDIAAI Dynamo!
Excited to share that ThunderAgent has been integrated into NVIDIA Dynamo as an experimental router for agentic workloads!
ThunderAgent was designed to schedule at the granularity of agent runs, making agentic serving/rl upto 4x faster!
Huge thanks to @0xishand , @KranenKyle , and the Dynamo team. They have been exceptionally efficient and proactive — the team had already started pushing this forward even before I officially joined @nvidia .
Looking forward to seeing ThunderAgent ideas further evolve within Dynamo. And thanks for the help from @togethercompute
Link: github.com/ai-dynamo/dynamo/…
@simran_s_arora @Chenfeng_X @_weilix @yinfang_chen
#AI #MLsys #Agent #Nvidia
Ben Athiwaratkun retweeted
Our inference stack, optimized for Blackwells, with a novel attention kernel and many new optimizations has started rolling out!
It's already charting on Artificial Analysis, eg: #1 speed and latency for @Kimi_Moonshot Kimi 2.6. #1 on latency on @MiniMax_AI, and miles ahead of other GPU endpoints.
artificialanalysis.ai/models…
artificialanalysis.ai/models…
Ben Athiwaratkun retweeted
LLM training is built on fast MatMuls. But many surrounding ops still run as memory-bound kernels.
CODA reparameterizes them to hide in the matmul’s shadow, fused into its epilogue before results leave the chip.
Bonus: LLMs can write fast CODA kernels too (approaching SoLs).
Thanks @SemiAnalysis_ for featuring our work on distribution-aware accelerated RL rollout
TLDR
> identify problems are tend to require more thinking & rollout tokens
> allocate more speculative decoding budget to problems that cause bottleneck
> use adaptive speculator to maintain high acceptance length due to policy changing over time (can be anything but in this case we show the use of suffix tree speculator which is lightweight and can adapt fast over narrower data distributions)
> speeding up rollout without changing quality due to the verification nature of spec decoding
We’ll be doing an oral presentation next week at MLSys
The long-tail distribution of rollout lengths causes one of the most critical inefficiencies in RL training.
To mitigate this issue, researchers proposed draft model techniques to boost throughput, e.g. Eagle, MTP, and DFlash.
Distribution-Aware Speculative Decoding for RL Training proposes a draft model that adapts to the distribution of token prediction and token length.
We believe adaptive draft model techniques is still ripe for innovation.
arxiv.org/abs/2511.13841
Authors: @tri_dao, @percyliang, @AlpayAriyak, @ben_athi, @yiying__zhang, @Chenfeng_X, @XiaoxiaWShirley, @QingyangWu1, @Ameen_ml (1/4)
We’re launching @JudgmentLabs today and announcing $32M in funding.
As AI agents take on more of the work that creates economic value, they generate massive amounts of production data: the clearest record of how they behave with users, software, and the real world.
Judgment builds infrastructure for improving AI agents from production data.
Ben Athiwaratkun retweeted
We push Prefill/Decode disaggregation beyond a single cluster: cross-datacenter + heterogeneous hardware, unlocking the potential for significantly lower cost per token.
This was previously blocked by KV cache transfer overhead. The key enabler is our hybrid model (Kimi Linear), which reduces KV cache size and makes cross-DC PD practical.
Validated on a 20x scaled-up Kimi Linear model:
✅ 1.54× throughput
✅ 64% ↓ P90 TTFT
→ Directly translating into lower token cost.
More in Prefill-as-a-Service: arxiv.org/html/2604.15039v1
Ben Athiwaratkun retweeted
🤔The more I studied diffusion language models, the more I came to appreciate the simplicity of autoregressive (AR) language models. AR models are trained to agree with what they generate, and their serving stacks are built to preserve that structure. DLMs often do neither: they lack introspective consistency, and high TPF does not necessarily translate into high real-world TPS.
We propose Introspective Diffusion Language Model (I-DLM), which unifies introspection and generation in a single pass:
1. 🧑🎓I-DLM brings introspective consistency to DLMs with only 5B training tokens, achieving AR-thinking-level quality.
2. 🚀 I-DLM carefully trades compute for higher TPF while converting that advantage into real TPS under high-concurrency serving.
📖Website: introspective-diffusion.gith…
⌨️Code: github.com/Introspective-Dif…
We're sharing an important progress in evolution via an efficient framework Squeeze Evolve
> not only improving capabilities, achieving SOTA @ 97.5% on ARC-AGI v2
> but also resulting in cost effectiveness (~3x lower than other methods)
> the key to this is using a multi-model framework
> multiple models bring diversity to sample solutions
> as well as allowing the use of smaller models on easier evolutionary steps
> verifier-free, which is crucial for tasks where verification is time-consuming
> a unified framework that combines all previous methods (Alpha Evolve, majority voting, self-refinement, RSA, and Mixture of Agents)
Super excited to Introduce our latest work: Squeeze Evolve.
We unify test-time scaling methods into one evolutionary framework — then orchestrate many models across it.
3x lower cost. 10x throughput. 97.5%(SoTA) on ARC-AGI-V2. No verifier required.
Framework: squeeze-evolve.github.io/
Overall we're heading towards a multi-agent future
> through evolution, organisms evolve to specialize
> energy/food are a constraints, so organisms evolve to be efficient
> brain capacity is also a constraint -- being experts in all tasks is hard and suboptimal for survival from an energy perspective
> the same will apply for LLMs
> strong generic models are used for the hardest tasks that require strong judgment and critical thinking
> sub-tasks can easily be allocated to less capable and more effective subagents
> specialized agents have much higher intelligence-per-joule for subtasks
> given global compute constraint, inference optimization such as multi-agent framework will be one of the biggest and most impactful research fields
New research on making speculators adapt to domain shift and keep your LLMs fast! Paper and code release in the thread.
@XiaoxiaWShirley @_junxiong_wang @ZhongzhuZhou @tri_dao @percyliang @Chenfeng_X
