A GPU has thousands of threads, and you would think the natural thing is to tell them all to pause at a line and wait for each other, the way you sync any group of workers. On a GPU, you cannot do that. There is no way to make every thread in a launch stop and wait at a global barrier, and the reason why is one of the most beautiful constraints in all of hardware. It comes down to two ideas: that closeness is speed, and that waiting can freeze the whole machine.
↳ Start with a distinction people blur constantly, because everything else depends on it. A block and an SM are not the same thing.
A block is software, just a number you pick in your code, a team of say 256 threads. You fully control it. An SM (Streaming Multiprocessor) is hardware, an actual physical room etched into the silicon with real math cores, real wires, and a real slab of fast memory inside it. You cannot change it, NVIDIA built it into the chip. So a block is a paper plan, and an SM is the building the plan runs in.
Why not just launch a million fully independent threads and let the hardware sort them out? Because independence is expensive in silicon. Every thread needs its own little scheduler.
A million independent threads would need a million little schedulers and a million separate pieces of state-tracking hardware just to keep them straight. All of that management machinery would eat the chip, leaving no room for the math units that are the whole point.
So the block is the compromise. You pin a team of threads to one physical room (the SM), and now a single scheduler manages the whole team and they share one pool of resources. The management stays cheap, and the silicon stays mostly math.
This is where closeness enters. Why can't half of a block run in one room and half in another? Because the fast scratchpad memory the team shares is physically inside the SM. It's a tiny slab of ultra-fast memory sitting millimeters from the cores, reachable through short local wires.
If you split a block across two SMs, half the threads would have to reach all the way across the chip to touch that shared scratchpad, which would be no faster than going to slow main memory. The entire speed advantage of sharing would vanish.
↳ So the rule is absolute: a block lands whole on one SM, so every thread in it can hit the shared scratchpad through local wires. This is locality, and locality is speed.
And the same logic explains why the in-block sync you do have only works inside a block. When you tell a block's threads to all wait for each other, that works because they are all in the same room, and the signal travels through fast local wires. One bell in one room, everyone hears it instantly.
Try to ring that bell across the whole factory, across dozens of separate SMs, and the signal has to crawl over the slow global buses connecting them. By the time it reached everyone, you would have spent as long as a slow memory access. The barrier was supposed to be fast, and distance destroyed it.
Why can't you just have a grid-wide barrier where every block in the launch waits for every other block? Because not all your blocks are actually running at the same time. You might launch a million threads, but the chip can only hold a fraction of them at once. So the hardware runs them in waves.
The first wave fills the SMs, the rest wait in a queue for room to open up. Now imagine the running wave hits a grid-wide barrier and stops, waiting for all the other blocks to arrive. But the other blocks are stuck in the queue, and they can't start until the running wave finishes and frees up the SMs.
↳ The Circular Wait:
The running wave waits for the queued blocks. The queued blocks wait for the running wave.
Each is waiting for the other. The machine freezes forever. This is a circular wait, the textbook recipe for deadlock.
And the obvious fix doesn't work either. Why not just pause the running wave, swap in a waiting block, let it reach the barrier, and swap back? Because a GPU cannot cheaply pause a block partway through.
A block holds its registers and scratchpad on the SM until it completely finishes, and there is no mechanism to yank it out mid-flight. Doing so would mean dumping all its registers out to slow main memory and reloading them later, which is exactly the expensive, locality-destroying thing the GPU is designed to avoid. So unlike a CPU, which casually pauses and resumes tasks all day, a GPU commits an SM to a block until that block is done. No take-backs.
↳ The rule emerges from physics, not from laziness. You cannot sync the whole grid because the blocks waiting in the queue can never reach the barrier while the running blocks are stuck waiting for them.
There is a clever exception that actually proves the rule rather than breaking it. There is a way to sync across blocks, but it comes with a hard condition: you may only launch as many blocks as can physically fit on the chip at the same time.
In other words, you are forced to launch exactly one wave, with no queue behind it. And once there is no queue, there is no circular wait, because every block is already resident on an SM, alive and able to reach the barrier. The deadlock vanishes, not by some trick, but because the dangerous situation—blocks stuck waiting outside—was eliminated by design. It is a strict confirmation of the principle: grid-wide sync is only safe when every participant fits in physical space at once.
A block is your software team, an SM is the physical room it runs in, and the block lands whole on one room so its threads can share fast local memory through short wires.
That same locality is why in-block sync is quick and why cross-chip sync would be slow. And grid-wide waiting is impossible in general because the chip runs work in waves, and a running wave waiting on a queued wave that can't start is a frozen circular wait.
The GPU didn't forget to add a global barrier.
It refused to, because the physics of closeness and the trap of waiting leave no other choice.