What is LLM Inference and How CUDA Graphs Improve It
Topics covered in this part:
- what exactly happens during LLM inference and how prefill and decode work
- what happens inside a single decode step and why it involves many small GPU kernel launches
- why LLM decode can become CPU / kernel-launch-overhead bound, especially during small-batch and latency-sensitive inference
- why CUDA Graphs are especially useful for the repetitive nature of token-by-token decoding
- what exactly gets captured inside a CUDA Graph and what gets replayed during each decode step
- how the execution structure can stay mostly the same while input tokens, model state, and KV-cache data continue to change
- how a mostly static CUDA Graph can work with dynamic LLM inference
- how the growing KV cache is handled using paged KV-cache management, block tables, and indirection
- where the actual performance improvement from CUDA Graphs comes from: reducing repeated CPU launch overhead and GPU idle gaps, not making the GPU kernels themselves faster
- how saving time on every decode step adds up across token generation, improving per-token latency and overall inference throughput