When the 32 threads of a bundle need to share data with each other, the obvious way is to write it into the shared scratchpad memory and have everyone read it back. That works, but it is heavier than it looks, involving a trip to a separate pool of memory, a synchronization barrier so nobody reads too early, and the risk of threads colliding on the same memory windows.
↳ There is a faster, almost magical alternative for threads within a single bundle, letting them read each other's private values directly, with no shared memory, no barrier, and no collision.
It is called a shuffle, and understanding it reveals something deep about why the threads in a bundle are special.
Start with the normal way and why it is clunky. Each thread has its own private working values, held in the fastest storage on the chip, its registers, which no other thread can normally see. So when thread five wants a number that thread twelve computed, it cannot just reach into thread twelve's private storage.
The usual solution is a shared bulletin board, the scratchpad. Thread twelve writes its number onto the board, then every thread pauses at a barrier until all the writing is done, and then thread five reads thread twelve's number off the board.
It works, but look at the cost. You went out to a separate memory pool, you forced everyone to stop and wait at the barrier, and if several threads read from the same region of the board they queue up and serialize. Three separate taxes just to pass a number between two threads that are already running side by side.
Now the question that motivates the whole trick: if they are all standing shoulder to shoulder marching in step, why should thread five have to leave a note on a distant bulletin board just to hand something to thread twelve? Why can't it just pass the note directly?
The 32 threads of a bundle are not really independent. They run in perfect lockstep, executing the same instruction at the same instant, physically right next to each other on the same piece of hardware.
That is exactly what a shuffle does. It is a single instruction that lets a thread reach directly into another thread's registers, within the same bundle, and read the value sitting there, with no detour through shared memory at all.
Thread five simply says, "give me the value that the thread twelve positions over from me is holding", and in one step it has it. No bulletin board, no writing, no reading back. The value moves straight from one thread's private register to another's. And because it happens as a single hardware instruction across the whole bundle at once, there is no separate barrier needed. The exchange is inherently synchronized, all 32 threads swapping values in the same instant.
And this raises the natural question of how that is even possible, when registers are supposed to be private?
The answer is that the 32 threads of a bundle physically share the same execution hardware and their registers sit together in the same register storage on that one unit. Because they move in lockstep and live in the same physical place, the hardware can wire a direct path between their registers, letting them peek at each other's values in a single step.
This is a privilege unique to threads within one bundle. Threads in different bundles, living on possibly different parts of the chip and not in lockstep, cannot do this at all, they are stuck with the slow bulletin board. The shuffle is a special power that comes precisely from the tight, lockstep intimacy of a single bundle.
So what is this actually good for? The classic use is collapsing 32 values down to one, like summing them. The old way had every thread write to the scratchpad, sync, read neighbors, sync again, repeatedly. With shuffles, the 32 threads pass their values directly to each other in a folding pattern, each step halving how many active values remain, all through direct register handoffs with no shared memory and no barriers at any point. What took many rounds of write-sync-read becomes a handful of clean shuffle steps.
Another use is broadcast, where one thread has a value everyone needs, and instead of it writing to the board for all to read, every thread just shuffles the value directly out of that one thread's register in a single step. These little bundle-wide exchanges, done through registers, are dramatically cheaper than routing everything through shared memory.
Now the honest limits, because the shuffle is powerful but narrow. It is confined entirely to a single bundle of 32 threads. It cannot pass a value to a thread in a different bundle, because that intimacy of shared hardware and lockstep only exists within one bundle.
So if you need to share data across the whole block of many bundles, you still have to fall back to the scratchpad bulletin board for that wider exchange. Shuffles handle the fast local sharing among 32 neighbors, and shared memory handles the broader sharing across the block, and fast code often uses shuffles for the tight inner exchanges and shared memory only where it genuinely must cross bundle boundaries.
So the whole picture. Threads normally have private registers, so sharing between them means a clunky trip through shared memory with a barrier and possible collisions. But the 32 threads of a bundle are special, running in lockstep on the same hardware with their registers physically together, which lets a shuffle instruction pass values straight from one thread's register to another's in a single synchronized step: no shared memory, no barrier, no collision.
It is perfect for collapsing or broadcasting values among the 32, and it is cheaper than the shared-memory route in every way, limited only by the fact that it cannot reach beyond the bundle.
The deeper idea is that the bundle of 32 is not just a scheduling detail.
It is a tightly bound unit whose members can talk to each other in ways that threads farther apart simply cannot,
and the fastest code exploits that closeness wherever it can.