Almost everyone learns one way to keep a GPU busy while its slow memory catches up: pile on more threads. When one group of threads stalls waiting for data, the chip switches to another group that is ready, and as long as you have enough groups, the cores never sit idle. This is true, and it's the story you hear everywhere. But it is only half the story, and the missing half explains a genuinely surprising fact: that a GPU running at a fraction of its supposed capacity can sometimes run at full speed anyway. There are two levers for hiding latency, not one, and understanding the second changes how you think about performance entirely.
↳ The secret is that you can hide latency within a single thread, not just across thread groups.
Start with the lever everyone knows. The slow part of a GPU is fetching data from far memory, which takes a long time. So when a group of threads asks for data, it stalls, and the chip fills the wait by running other groups that already have their data. The more groups you have resident and ready, the more the chip can switch around and stay busy. How full this pool of resident groups is has a name (occupancy), and people treat filling it up as the whole goal of performance tuning. More groups, more coverage, more hiding. This is real, and it's the first lever.
But here's the trap in thinking it's the only one. The long wait is genuinely long, and the pool of groups is genuinely limited by the chip's fixed resources. Do the honest accounting and you find that filling the pool to its absolute maximum still often isn't enough on its own to cover the full length of the wait. There simply aren't enough groups allowed at once to paper over how long the memory takes. So if more groups were the only lever, the chip could never fully hide its latency, and yet it clearly does. Something else must be helping.
That something is the second lever, and it lives inside a single group of threads rather than across many. So far we imagined each group firing off one memory request and then stalling until it comes back. But what if a single group fires off several independent requests before waiting for any of them? It asks for the first piece of data, and instead of immediately waiting, it also asks for the second, third, and fourth, all of which it will need but none of which depend on each other. Now all those requests are traveling to memory at the same time, overlapping, and the group only waits once for all of them to arrive together rather than paying the full wait four separate times. One group, but several things in flight at once.
So the question this raises: if a single group can keep several requests in flight on its own, does it still need a huge pool of other groups to hide the latency?
And the answer is no, and this is the surprise. The total amount of work in flight is what matters for hiding the wait, and that total is the number of groups times how many requests each group juggles. So you can reach the amount you need either way: with many groups each doing one thing, or with fewer groups each doing several things at once. The two levers multiply to give you the coverage you need, so pulling harder on one lets you pull less on the other.
Which means the thing everyone treats as the master dial, how full the pool of groups is, is not actually the goal. It is one way to buy coverage, not the only way. And here is the counterintuitive consequence: a kernel running with only a modest pool of groups, well below the chip's maximum, can completely hide its latency and hit peak speed, as long as each of those groups is juggling several independent operations at once. It looks underutilized by the usual measure, the pool is half empty, and yet it runs flat out, because the second lever made up the difference. Low on the metric everyone watches, full speed in reality.
So how do you actually pull this second lever, since it doesn't come for free? It comes from how you write the inner work of each thread. Instead of writing code that does one thing, waits for it, does the next thing, waits for it, you write code that launches several independent pieces of work up front and only then starts consuming their results. Often this means unrolling a loop so that a few iterations' worth of independent loads all go out together, or arranging the math so a thread holds several running results at once rather than one. The code deliberately creates independent work within a single thread, so the hardware has several things it can have in flight simultaneously. It's a code-structure decision, not a launch-configuration one.
And this connects to a tension people hit constantly without understanding it. Pulling this second lever costs each thread more of the chip's scratch resources, because holding several things in flight means holding several partial values at once. And using more of those resources per thread means fewer groups can be resident, shrinking the first lever's pool. So the two levers are not just additive alternatives, they trade against each other through the same limited budget. Push hard on per-thread parallelism and you naturally get fewer resident groups, which is exactly why a low-pool kernel can still be fast: it spent its budget on the second lever instead of the first. The mistake is to force the pool full by starving each thread, which can leave you worse off than a leaner-looking kernel that hides latency from within.
Hiding memory latency needs enough independent work in flight to cover the long wait, and there are two ways to supply it: resident thread groups (lever one) and intra-thread instruction parallelism (lever two).
These two levers multiply, so you can hit full speed with a big pool of simple groups or a small pool of busy ones, which is why low occupancy can secretly run at 100% throughput. Because registers are finite, the two levers trade off against each other.
The real target is total independent work in flight...
the best code pulls the quieter lever within each thread.