Why occupancy matters

*worked out from first principles*

There's a number GPU programmers obsess over called occupancy, basically how full the chip is with active work. And the trap almost everyone falls into is thinking more of it is always better. Fill the chip to 100 percent and surely it runs fastest.

The truth is messier, and getting why is one of the sharper things you can understand about how a GPU actually performs.

The Backup Bench: Hiding Latency

Start with how a GPU deals with slow memory, because that decision shapes everything else.

A CPU and a GPU face the same problem: data from memory takes ages to arrive. But they solve it in opposite ways. A CPU tries to minimize the wait. It spends a lot of silicon on big caches so the data is already sitting close by when it's needed.

A GPU refuses to spend silicon that way. Instead it hides the wait. When one warp asks memory for data that won't land for hundreds of cycles, the GPU doesn't sit there. It shoves that warp aside and instantly swaps in another warp that has work ready to go. The math units never stop; the GPU just keeps changing which worker is at the bench.

That's latency hiding, and it's the whole reason a GPU wants lots of warps loaded up. The spare warps are the backup bench that keeps the chip busy while others wait on data.

↳ Now you can see why high occupancy looks good. More warps loaded means more backup, more ready work to switch to, more chances to hide the waiting. So far the instinct holds.

The Trap of 100% Occupancy

Here's where it breaks. Picture a restaurant with one very fast chef. The chef is the math unit, the customers waiting to order are the warps, and the supplier bringing potatoes from the market is memory.

Now fill the place completely, every seat taken, 100 percent occupancy. Looks perfect, tons of backup. But the market is out of potatoes. The first customer says "I want a burger but we have to wait for potatoes." Fine, the chef moves to the next customer, who says the same thing. And the next. Every single customer is waiting on the exact same delivery.

The chef has a hundred people in the room and not one of them can be served, because they all need the one thing that hasn't arrived.

That's a GPU stalled on memory. When every warp on the bench is waiting on the same slow memory, the scheduler has nobody to switch to. There is no ready work anywhere. So the math unit sits idle even at 100 percent occupancy. The chip was full, and full bought you nothing.

Occupancy was never the goal. It was only ever a tool to hide latency, and here there's no latency left to hide, just a universal wait.

The Fix: Do More with Less

So what actually fixes it? You flip the usual advice and do less, per a strange logic.

Instead of a thousand threads each fetching one number, you use far fewer threads, say a quarter as many, and have each one fetch four numbers at once and work on all of them. Occupancy drops, maybe to 25 percent, which sounds like a downgrade. But two good things happen.

• First, the traffic to memory drops. A thousand separate little requests jamming the memory controller becomes a few hundred bigger ones, so the bus is less congested and data comes back sooner.
• Second, each thread now has independent work to chew on. While the first of its four numbers is still in transit from memory, the thread can already be doing math on instructions queued for the others. It keeps its own pipeline busy instead of just waiting.

This is instruction-level parallelism, and it means a thread hides latency inside itself rather than relying on the scheduler to find another warp. So lower occupancy, but higher actual performance, because the work got denser.

Hardware Limits: Registers & Shared Memory

There's a second reason occupancy isn't something you simply crank to the max. You don't fully control it; the hardware does, based on resources.

Each Streaming Multiprocessor (SM) has a fixed pool of registers and a fixed amount of fast on-chip shared memory, and the threads have to split those pools.

Take registers. Say an SM has a fixed budget of them. If each thread uses few registers, loads of threads fit and occupancy is high. But the moment your code gets complex and each thread needs more variables, each thread eats more registers, and suddenly only half as many threads fit on the SM. The rest wait in line. Your occupancy just got cut in half, not by choice but because you ran out of register space.

The same trap exists with shared memory. If a block reserves a big chunk of the on-chip scratchpad, then even if registers are free, no new block can move onto the SM because there's simply no room left. You ran out of space, so occupancy fell.