Why LLM Inference Frameworks Use torch.compile
Topics covered in this part:
- Why LLM inference is a perfect candidate for compilation
- Training vs Inference
- Autoregressive generation and repeated decoder execution
- Compilation cost amortization
- PyTorch Eager Execution
- How eager execution works internally (Python → Dispatcher → CUDA → Python)
- Why eager execution limits compiler optimizations
- What torch.compile actually changes
- High-level torch.compile pipeline (TorchDynamo → FX Graph → TorchInductor → Triton/C++)
- Why inference frameworks like vLLM and SGLang use torch.compile