Why GPU programs actually slow down

*worked out from first principles*

Your GPU is rated for a hundred trillion operations a second. You run your code and it delivers five. Is the chip broken? Do you need cleverer math? Almost never. A GPU is a supercar, and a supercar is rarely slow because the engine is weak. It's slow because it's stuck in traffic or out of fuel. Beginners respond to a slow kernel by tuning the arithmetic. Experts know the real fight is almost always somewhere else, in how data moves and how work gets scheduled. There are six classic ways a GPU ends up starving, and knowing all six turns a mysterious slowdown into a diagnosis.

↳ The thing to hold onto first is what unites all six. Every one of them is a different way of failing to keep the compute cores fed with useful work. The cores themselves are almost never the problem. So each bottleneck is really a story about data or scheduling failing to deliver, and the symptom is always the same: cores sitting idle while the clock runs.

1. Transfer Overhead

The first horseman is transfer overhead, the worst of all. This is when your program spends nearly all its time shipping data back and forth between the main computer and the GPU across the narrow bridge that connects them, for a sliver of actual math. You copy data over, do one tiny operation, copy it back, and the round trip dwarfs the work. The cores might be busy eight percent of the time.

The cure: Stop the ping-pong. Keep the data living on the GPU across many operations, and batch work into big transfers instead of many tiny ones, so the crossing cost is paid rarely and amortized.

2. Memory-Bound

The second is being memory-bound, and this one surprises people who assume the math is the point. The deciding number is how much arithmetic you do per byte you fetch. For something like adding two arrays, you move a dozen bytes to do a single addition. The cores finish that instantly and then sit waiting for the memory system to deliver the next batch, which it simply cannot do fast enough. The cores starve not because they are slow but because the data pipe is the limit.

The fix: Do more math per byte, by pulling a piece of data into fast on-chip memory once and reusing it many times before letting it go, so each expensive fetch pays off across many operations.

3. Low Occupancy

The third is low occupancy, which attacks the GPU's core survival trick. The way a GPU hides the long wait for memory is by always having other bundles of threads ready to run while some are stalled. Occupancy is how full that reservoir of ready bundles is. When it's low, a bundle stalls on memory and there is no backup to swap in, so the cores go idle waiting. And the usual cause is greed: each thread grabbing too many registers or each block hogging too much fast memory, which leaves room for too few bundles to be resident at once.

The fix: Trim that per-thread appetite so more bundles fit and the scheduler always has a backup ready.

4. Branch Divergence

The fourth is branch divergence, which strikes at how a bundle executes. The 32 threads of a bundle move in lockstep, all running the same instruction together. Put a branch in the code where some threads go one way and others go the other, and the hardware cannot run both paths at once. It runs the first path with half the threads active and the rest frozen, then runs the other path with the halves swapped. What should have been one pass becomes two, and the more tangled the branching, the worse it gets.

The fix: Arrange things so threads in the same bundle tend to take the same path, for instance by grouping similar data together, or by replacing branches with arithmetic that all threads run identically.

5. Synchronization Cost

The fifth is synchronization cost, the price of making threads wait for each other. A barrier that makes every thread in a block pause until all arrive is necessary sometimes, but it is not free. When you hit one, every thread waits for the slowest straggler, so if one thread is delayed, hundreds sit idle for it. Sprinkle barriers everywhere just to be safe and you turn a smooth parallel flow into stop-and-go traffic.

The fix: Discipline. Sync only exactly where data genuinely crosses between threads, use the lighter bundle-level sync when you only need a bundle to agree, and never scatter barriers as a superstition.

6. Poor Coalescing

The sixth is poor coalescing, the most common and most invisible killer of all. When a bundle reads memory, the hardware delivers it in fixed-size boxes. If the 32 threads read neighboring slots, they all fit in one box, one fetch. If they read scattered slots, each triggers its own full box fetch, most of which is thrown away, and the bundle drowns in wasteful transactions. The cores stall waiting for a flood of nearly-empty deliveries.

The fix: Arrange data so that neighboring threads read neighboring addresses, choosing layouts where the value all threads want at once sits packed together.