The fast on-chip memory that GPU threads share is often called a scratchpad, and it is praised for being roughly a hundred times quicker than main memory.
↳ But that speed comes with a hidden rule, and if you break it, that same fast memory quietly slows down by up to thirty-two times, turning your carefully optimized code into a crawl.
The culprit is something called a bank conflict, and it is one of the most common invisible performance killers in GPU code. Understanding it means understanding that fast memory is only fast if all your threads access it in the right shape.
Start with why this fast memory is fast in the first place. The reason it can serve a whole bundle of 32 threads almost instantly is that it is not one single door that everyone queues at. It is physically built as 32 separate doors, or windows, side by side, each able to hand out data independently at the same moment.
So when the 32 threads of a bundle each want a value, the ideal case is that all 32 windows open at once, each serving one thread, and the entire bundle gets its data in a single tick. Thirty-two requests satisfied in parallel through thirty-two independent windows. That parallelism across the windows is the entire source of the speed.
Now the catch, which follows directly from that design. The memory is spread across those 32 windows in a repeating pattern, consecutive slots of memory assigned to consecutive windows, wrapping around. So slot zero belongs to window zero, slot one to window one, and so on, and after slot thirty-one it wraps back so slot thirty-two belongs to window zero again. Each window is responsible for its own set of slots.
And here is the rule: a single window can only serve one thread at a time. If two threads in the bundle happen to want slots that live behind the same window, that window cannot open twice at once. It has to serve the first thread, then the second, one after the other. Those two threads are now serialized, waiting in line at the same window while the other windows sit idle.
So picture the disaster case. Suppose every one of the 32 threads wants a slot that happens to map to the very same window. Now instead of 32 windows opening in parallel, you have one window serving 32 threads in a slow single-file line, one after another, while the other 31 windows do nothing. The access that should have taken a single tick now takes thirty-two ticks. You got the worst possible slowdown, and you were using the fast memory the whole time. It wasn't slow because of the memory, it was slow because your threads all crowded the same window. This is a bank conflict, two or more threads in a bundle colliding on the same window, forcing serialization.
So what actually causes threads to crowd one window?
It comes down to the spacing between the slots your threads access, the stride. If the 32 threads read 32 consecutive slots, each lands on a different window, perfect parallelism, no conflict. But suppose each thread reads a slot spaced out by a stride that happens to line up with the number of windows. Then thread after thread lands on the same window, because the wrapping pattern brings them all back to the same place.
A very common way this happens is walking down a column of a two-dimensional grid whose width matches the window count, where every element in that column sits behind the same window. So a completely natural access pattern, reading down a column, can accidentally funnel the whole bundle into one window and trigger the maximum slowdown, without you ever seeing anything obviously wrong in the code.
Now a fair objection, because it seems like sharing a window should always be the problem. What if all 32 threads want the exact same slot, not just the same window but the identical value? Isn't that the worst conflict of all, 32 threads on one window?
Surprisingly, no, and this is a neat exception. When all the threads want the very same value, the hardware doesn't make them line up. It simply reads that one value once and broadcasts it to all of them simultaneously, in a single tick.
The conflict only happens when threads want different slots that unfortunately share a window. Wanting the identical slot is free, served to everyone at once. So it is not sharing a window that hurts, it's specifically different addresses colliding on the same window. Same address, broadcast, fast. Different addresses, same window, serialized, slow.
So how do you fix a conflict once you find one? The classic trick is delightfully simple: you pad the memory. If reading down a column is funneling everyone onto one window because the grid's width lines up badly with the window count, you just make each row of the grid one slot wider than it needs to be.
That little extra padding shifts where each row starts, so that the elements of a column now land on different windows instead of the same one. You waste a tiny bit of memory on padding you never use, and in exchange the whole bundle spreads across all the windows and reads in a single tick again. It's one of the highest-value-for-effort fixes in GPU programming, a sliver of wasted space buying back a thirty-two-fold slowdown.
So the whole picture. The shared scratchpad is fast because it is built as many independent windows that can all serve threads at once, but each window can only serve one thread at a time. When threads in a bundle want different slots that map to the same window, they collide and get serialized, and in the worst case a full bundle crowding one window runs thirty-two times slower, all while appearing to use the fast memory.
The collisions come from the stride of your access lining up badly with the number of windows, classically when marching down a column. The one exception is that all threads wanting the identical value is free, because the hardware broadcasts it. And the standard cure is to pad your data so accesses spread back across all the windows.
The deeper lesson is a familiar one on this hardware: fast is never automatic.
Even the fastest memory on the chip is only fast if you approach it in the shape it was built for.