FlashAttention-4: Algorithm and Kernel Pipelining
FlashAttention-4 introduces a new algorithm and kernel pipelining design that addresses asymmetric hardware scaling, improving performance on modern GPU architectures by better managing memory and compute resources.
Background
- FlashAttention is a family of algorithms that make Transformer models (like GPT, Llama, Claude) much faster by reducing how often the GPU reads and writes memory during attention computations — the most expensive part of training and running these models.
- FlashAttention-1 and 2 tiled the computation so it stayed in fast on-chip memory; FlashAttention-3 added FP8 (low-precision) support for even faster throughput.
- FlashAttention-4 introduces "kernel pipelining": overlapping parts of the attention calculation (the "algorithm" side) with the movement of data in GPU memory (the "kernel" side), treating them as one co-designed unit rather than separate steps.
- This matters because GPU memory bandwidth is not scaling as fast as GPU compute power ("asymmetric hardware scaling"), making data movement the new bottleneck. FlashAttention-4's pipelining hides that latency, delivering speedups for both training and inference of large language models.
- Colfax International is a research and engineering firm specializing in high-performance computing, GPU optimization, and AI infrastructure.