@kwatra
Joined January 2009
  • Tweets19
  • Following74
  • Followers45
3Media
Long-context inference is hitting a wall. 🛑 As context grows, Attention becomes the villain. Why? • Decode: Attention scales linearly (O(N)), while the rest of the model stays constant (O(1)). • Prefill: Attention explodes quadratically(O(N²)). Can we do better?(1/9)
Graph showing percentage of attention and other operations in total decode time.

ALT Graph showing percentage of attention and other operations in total decode time.

1
1
2
4
670
Evaluation of LLM serving systems is tricky because several factors influence performance (prefill length, decode length, parallelization) and there are multiple metrics we care about (throughput, ttft, tpot/tbt). We identify common pitfalls and a checklist to avoid them.
Amey Agrawal
@agrawalamey1218 Jul 2025
The bitter lesson of AI infra: The hardest part about building faster LLM inference systems is not designing the systems, but rather it is evaluating if the system is actually faster! 🤔 This graph from a recent top systems venue paper about long-context serving shows average normalized input token latency for a trace with both short and 100K+ token requests. System X looks like a clear win: lower normalized latency and higher request rates. But normalized metrics can obscure the actual user experience: at those rates, long inputs see >2hr delays to the first token! Let’s do the math!🧮
2
7
545
The bitter lesson of AI infra: The hardest part about building faster LLM inference systems is not designing the systems, but rather it is evaluating if the system is actually faster! 🤔 This graph from a recent top systems venue paper about long-context serving shows average normalized input token latency for a trace with both short and 100K+ token requests. System X looks like a clear win: lower normalized latency and higher request rates. But normalized metrics can obscure the actual user experience: at those rates, long inputs see >2hr delays to the first token! Let’s do the math!🧮
1
9
1
23
2,120
We have released the source code and benchmarks of TokenWeave. TokenWeave speeds up distributed LLM inference via compute–communication overlap and fused AllReduce, RMSNorm, and residual addition. Code: github.com/microsoft/tokenwe… Paper: arxiv.org/pdf/2505.11329 Try it out!

GitHub - microsoft/tokenweave: Accepted to MLSys 2026

Accepted to MLSys 2026. Contribute to microsoft/tokenweave development by creating an account on GitHub.

github.com
Ram Ramjee@ramaramjee20 May 2025
TokenWeave is the first system that almost fully hides the ~20% communication cost during inference of LLMs that are sharded in a tensor-parallel manner on H100 DGXs. Check out the thread/paper below!
1
2
2
7
3,777
TokenWeave – Efficient Compute-Communication Overlap for Distributed LLM Inference. Why? Even with highspeed NVLink on H100 DGX, communication overhead for distributed LLM inference can be > 20 %! Can we recover this overhead? (1/10)
1
5
1
17
1,464
more replies
Small token batches enable chunked-prefill schedulers (e.g., Sarathi). (9/10)
1
2
151
Dive into the details → arxiv.org/pdf/2505.11329. Code on the way—stay tuned! (10/10)
3
148
🚨 Are LLM compression methods (𝘲𝘶𝘢𝘯𝘵𝘪𝘻𝘢𝘵𝘪𝘰𝘯, 𝘱𝘳𝘶𝘯𝘪𝘯𝘨, 𝘦𝘢𝘳𝘭𝘺 𝘦𝘹𝘪𝘵) too good to be true and are existing eval metrics sufficient? We've looked into it in our latest research at @MSFTResearch 🧵 (1/n) arxiv.org/abs/2407.09141

Accuracy is Not All You Need

When Large Language Models (LLMs) are compressed using techniques such as quantization, the predominant way to demonstrate the validity of such techniques is by measuring the model's accuracy on...

arxiv.org
2
7
4
19
5,358
🚀 Introducing Metron: Redefining LLM Serving Benchmarks! 📊 Tired of misleading metrics for LLM performance? Our new paper introduces a holistic framework that captures what really matters - the user experience! 🧠💬 github.com/project-metron/me… #LLM #AI #Benchmark

GitHub - project-etalon/etalon: LLM Serving Performance Evaluation Harness

LLM Serving Performance Evaluation Harness. Contribute to project-etalon/etalon development by creating an account on GitHub.

github.com
2
15
5
34
6,696
Did you ever feel that @chatgpt is done generating your response and then suddenly a burst of tokens show up? This happens when the serving system is prioritizing someone else’s request before generating your response. But why? well to reduce cost. 🧵
1
5
3
26
7,445
Ever wondered why @OpenAI charges 2x price for output tokens compared to input? Turns out that an output token can be up to 200x more compute time than an input token. Why? We explored this phenomenon during my internship at @MSFTResearch. 🧵
7
46
2
373
78,093
something
2
Load more