we launched the most comprehensive ai performance engineering repo in the world and we're posting every single resource follow and save to keep up with the series. links in thread 🧵 part 10: CUDA Programming Guide making an LLM faster means understanding where execution is waiting. an attention kernel can wait for its next tile of data, while the serving runtime can leave the GPU waiting for the next launch. NVIDIA's CUDA Programming Guide explains the mechanisms behind those waits. for a modern AI performance engineer working in PyTorch, Triton, or CUDA, it helps with: - tuning attention, matmul, and normalization kernels. the SIMT and CUDA Tile chapters connect layouts and data reuse to GPU execution. the resource guidance explains why larger tiles or more fusion can introduce register spills. - overlapping data movement with computation. asynchronous copies, TMA, and producer/consumer pipelines explain how supported GPUs can load the next tile while processing the current one, with the synchronization required to reuse buffers safely. - managing growing KV caches. the virtual memory chapter explicitly connects its allocation model to LLM serving: reserve an address range, then map additional physical memory into it as the cache grows, without relocating existing data. - reducing overhead between inference kernels. CUDA Graphs reduce repeated CPU launch setup, while stream-ordered allocation avoids broad synchronization around temporary memory. these matter when launch and allocation costs contribute to request latency. - diagnosing stalls across GPUs. the guide explains peer access and communication dependencies, including how a compute kernel's completion fence can wait on unrelated NCCL writes. memory synchronization domains can reduce that interference on supported hardware. - evaluating CPU memory offload. the memory chapters explain how residency, migration, and interconnect traffic affect access cost. that helps assess the latency and capacity tradeoffs of keeping model data in CPU memory on supported systems. - validating an optimization before deploying it. the appendices explain how reduction order, accumulation precision, and fast math can change results, and which hardware capabilities and resource limits a kernel depends on. when a training step or inference request slows down, these details help narrow the next experiment: change the kernel, move less data, adjust allocation, or remove an unnecessary wait.