Why NVIDIA made CUDA asynchronous

*worked out from first principles*

The Assembly Line

The obvious way to run a big job on the GPU is three steps in a row. Copy the data over, run the work, copy the results back. It works, and it wastes most of your machine. Because while the data is copying in, the compute cores sit idle. While the cores compute, the copy machinery sit idle. While the results copy out, everything else sits idle. You paid for all these parts and you use them one at a time, taking turns, like a factory where only one station is ever running. The fix is to stop thinking in steps and start thinking like an assembly line, and understanding how that works reveals a surprising limit hiding underneath the word asynchronous.

↳ Start with the hardware fact that makes it possible. A GPU does not have one worker, it has three independent ones that can all run at the same moment. There is an engine that copies data from the CPU's memory into the GPU's memory. There is the compute machinery that actually runs your work. And there is a third engine that copies results back from the GPU to the CPU. These are separate circuits. That means, at a single instant, data can be flowing in, other data can be computing, and still other results can be flowing out, all at once, if you arrange the work right.

So here's the objection. How can compute and copy-in run at the same time? Doesn't the work need its data before it can compute on it? Yes, but not the same data. The trick is to break the job into chunks. While the compute engine is chewing on chunk one, the copy-in engine is already fetching chunk two, and the copy-out engine is shipping chunk zero's finished result back home. Each chunk still flows through all three stages in order: copy in, then compute, then copy out. But at any given moment the three engines are each working on a different chunk. It's exactly a factory line. As one car moves to painting, the next moves to assembly, and a third rolls in the door. Nobody waits.

And the payoff is large. In the naive take-turns version, if each of the three stages takes some time, every chunk costs you all three added together. In the overlapped line, once the pipeline is full, the three stages happen in parallel, so each chunk effectively costs only the single longest stage, not the sum. That is close to a three-fold speedup, and it comes not from faster math but purely from refusing to let expensive hardware sit idle.

Fire, Forget, and the Saturated Queue

Now who orchestrates this? You might think the CPU has to babysit every handoff, but it doesn't, and this is where the earlier idea of fire-and-forget comes back. The CPU uses queues, in-order lines of GPU commands. It dumps a sequence of orders into a queue: copy this chunk, compute this chunk, copy this result back, and immediately moves on without waiting. It can throw dozens of commands into the queues in a flash, and the GPU drains them at its own pace, keeping all three engines fed. The CPU is a manager rapidly scribbling tickets, not a worker standing over each machine.

But here is the twist that catches people who fall in love with the word asynchronous. If the CPU never waits, does that mean a loop launching a billion pieces of work would never stall the CPU at all? It feels like it should. But no, the CPU will eventually freeze. Because the command queue is not infinite. It is a bounded buffer that holds only so many pending commands. The CPU can fire commands faster than the GPU can complete them, so the queue fills up. And the moment it is full, the next launch has nowhere to go, so it blocks, and the CPU stalls right there, waiting for the GPU to finish one item and free a slot. So asynchronous does not mean unlimited. It means the CPU runs ahead freely only up to the depth of the queue. Once the buffer saturates, the backpressure quietly forces the CPU and GPU back into lockstep. The decoupling is real, but bounded.

The Traps of Overlap

So you understand the pipeline, and you understand the queues. But getting them to actually overlap requires navigating two specific traps.

The first trap is the single queue. CUDA calls its queues streams. If you just launch copy, compute, copy in a loop without specifying a stream, everything goes into Stream 0, the default queue. And a single queue is strictly in-order. Command two cannot start until command one finishes. If chunk one's compute is in the queue behind chunk zero's copy-out, the GPU will not start computing chunk one, even though the compute engine is idle, because it must finish chunk zero's copy first. A single queue completely destroys the assembly line, collapsing everything back into taking turns. To get overlap, you must create multiple streams and put chunk zero in stream A, chunk one in stream B, and so on. Only then will the GPU scheduler see independent lines of work and schedule them on separate engines at the same time. Independent lines are what let independent engines run in parallel.

Second, the copy engine can only run truly independently if the CPU's memory is locked in place. Normally the operating system is free to quietly relocate your data in memory whenever it likes. But the copy engine reaches directly into CPU memory on its own, without the CPU's help, and if the data moved out from under it, that would break. So for the async copy to work, you have to pin the memory, telling the system to lock this data at a fixed spot and never move it. Only then can the copy engine pull from it directly and independently while the CPU goes off doing other things. Without pinning, the copy quietly falls back to the slow, blocking behavior and your overlap silently disappears.