Modern GPU Programming for MLSys Book
This book provides an introduction to modern GPU programming techniques with a focus on machine learning systems. It covers GPU architecture, CUDA, and optimization strategies relevant for building efficient ML models and frameworks.
Background
- This is an online book (free, in progress) teaching GPU programming specifically for people building machine learning systems — not for general-purpose GPU coding.
- MLsys (Machine Learning Systems) is the engineering discipline behind training and deploying large neural networks; its practitioners write kernels (low-level GPU code) to make models run faster or use less memory.
- The book covers modern GPU concepts like the CUDA programming model, tensor cores (specialized hardware for matrix math), FlashAttention (a now-standard attention algorithm that avoids slow memory transfers), and quantized inference (running models with lower-precision numbers to save memory).
- Author relates these to recent ML advances (e.g., Llama, a widely used open-source LLM family) so readers can connect low-level GPU tricks to real models they've heard of.