murali krishnan

You hit "Send" on an LLM API call. ~400 milliseconds later you get a response. Ever wonder what happened in between? 14+ infrastructure layers. Billions of matrix multiplications. Thousands of dollars in GPU compute per hour. This applies to every major provider — OpenAI, Anthropic, Google, all of them. Here's the full journey: 𝗦𝘁𝗲𝗽 𝟭: 𝗔𝗣𝗜 𝗚𝗮𝘁𝗲𝘄𝗮𝘆 (~5ms) Your request doesn't go straight to a model. → TLS termination → API key validation → Rate limiter checks tokens-per-minute and requests-per-minute → Request schema validation → Billing meter starts ticking That 429 error you sometimes get? It dies right here. 𝗦𝘁𝗲𝗽 𝟮: 𝗟𝗼𝗮𝗱 𝗕𝗮𝗹𝗮𝗻𝗰𝗲𝗿 (~2ms) Your request gets routed to one of many GPU clusters. → Geographic routing to nearest data center → Least-connections algorithm picks the cluster → Health checks running continuously This is why latency varies between identical calls. 𝗦𝘁𝗲𝗽 𝟯: 𝗧𝗼𝗸𝗲𝗻𝗶𝘇𝗮𝘁𝗶𝗼𝗻 (~3ms) Your text gets converted to numbers. → BPE, SentencePiece, or WordPiece depending on the provider → Each token is roughly 4 characters → Context window limit checked here Token count equals your cost. This is where the meter runs. 𝗦𝘁𝗲𝗽 𝟰: 𝗠𝗼𝗱𝗲𝗹 𝗥𝗼𝘂𝘁𝗲𝗿 (~1ms) The layer nobody talks about. → Large model requests route to heavy multi-GPU clusters → Small model requests route to optimized single-GPU clusters → Embedding requests route to dedicated clusters → Queue management during peak traffic Every provider with multiple models has this layer. 𝗦𝘁𝗲𝗽 𝟱: 𝗜𝗻𝗳𝗲𝗿𝗲𝗻𝗰𝗲 (~300-800ms) This is 95% of your wait time. Two phases: 𝗣𝗿𝗲𝗳𝗶𝗹𝗹: → All input tokens processed in parallel → Attention scores computed across Query and Key matrices → KV Cache generated and stored in GPU HBM memory → This is why long prompts have higher time-to-first-token 𝗗𝗲𝗰𝗼𝗱𝗲: → One token generated per forward pass → KV Cache reused so past tokens aren't recomputed → Temperature and top_p sampling happens at this step → Each token sent immediately if streaming is on This is the fundamental reason streaming exists — tokens are generated one at a time. The hardware doing this: → A100 / H100 / H200 GPUs with 80GB+ HBM → Model weights split across multiple GPUs via tensor parallelism → Multiple user requests batched together for throughput → Flash Attention and GQA for memory efficiency This is why GPU compute costs $2-3 per hour. 𝗦𝘁𝗲𝗽 𝟲: 𝗣𝗼𝘀𝘁-𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 (~5ms) → Token IDs converted back to readable text → Safety classifier runs on the output → Stop sequences checked → Response packaged as JSON 𝗦𝘁𝗲𝗽 𝟳: 𝗕𝗶𝗹𝗹𝗶𝗻𝗴 & 𝗥𝗲𝘀𝗽𝗼𝗻𝘀𝗲 → Input tokens × price per 1K → Output tokens × price per 1K (usually 3-5x more expensive) → Prompt caching reduces cost at most providers now None of this is specific to one provider. This is how LLM inference works at scale — whether you're calling OpenAI, Anthropic, Google, or anyone else.