Decoupling Compute and Memory for Async GPUs
An open-source project introduces VDCores, a new programming model for NVIDIA GPUs that decouples compute and memory, enabling asynchronous memory operations. It reports a 12% performance improvement over existing state-of-the-art and a 67% reduction in kernel code.
Background
- Modern NVIDIA GPUs (Hopper generation and later) support asynchronous execution — they can launch compute work and memory transfers that overlap in time, rather than stalling while data moves. Most existing GPU programming models don't let developers easily exploit this, so the hardware's potential goes untapped.
- The linked project (VDCores) proposes a new programming model that explicitly separates ("decouples") compute operations from memory operations. This lets the GPU schedule each independently, improving utilization.
- The project claims a 12% performance gain over current state-of-the-art approaches, while requiring 67% less boilerplate kernel code — meaning developers write simpler programs that run faster.
- The paper is available on arXiv (2505.03190); the code is open-source. This is aimed at GPU programmers, compiler researchers, and anyone working on high-performance computing or AI inference workloads where GPU memory bottlenecks are a known pain point.