Understanding Latency Hiding on GPUs [pdf]
This technical report examines latency hiding techniques on Graphics Processing Units (GPUs), analyzing how GPUs hide memory and instruction latency through massive multithreading and warp scheduling. The authors present a detailed study of GPU pipeline behavior and quantify the effectiveness of different latency hiding mechanisms across various workloads.
Background
- GPUs (Graphics Processing Units) were originally designed for rendering graphics but are now widely used for general-purpose computing (GPGPU), especially in machine learning and scientific simulation.
- "Latency hiding" is a core GPU design concept: when one group of threads is waiting for data (e.g., from memory), the GPU quickly switches to another group of threads that is ready to compute, keeping the hardware busy.
- This paper (from UC Berkeley EECS, 2016) provides a formal framework to analyze how well different GPU architectures hide memory latency. It introduces the concept of "occupancy" — how many active threads a GPU maintains to keep its execution units fed.
- Understanding this is key to writing efficient GPU code: if you don't have enough concurrent threads, the GPU stalls waiting for data, wasting its massive parallel throughput potential.