FlashAttention-4: Algorithm and Kernel Pipelining
FlashAttention-4 introduces a co-designed algorithm and kernel pipelining approach to improve attention computation on hardware with asymmetric scaling properties, enhancing efficiency and throughput on modern accelerators.
Background
FlashAttention is a series of algorithms that make transformer models (like GPT, Claude, and Llama) run far more efficiently by reducing how much data must move between a GPU's fast on-chip memory and its slower main memory. FlashAttention-1/2/3 progressively improved this for standard GPU setups. FlashAttention-4, introduced by Tri Dao (now at Princeton) and the FlashAttention team, tackles a new reality: modern GPUs (e.g., NVIDIA Blackwell B200) have asymmetric hardware scaling, meaning compute speed grows faster than memory speed, creating a bottleneck. The paper proposes "kernel pipelining" — overlapping memory transfers with computation across different "warps" (GPU thread groups) — to keep the math units fed despite the memory lag. This is cutting-edge infrastructure work: most LLM inference and training speedups in practice come from attention optimizations like this, so FlashAttention-4 directly impacts how fast and cheap large models run on next-gen hardware.