Why shared memory is fast

*the scratchpad and the art of data reuse*

The slowest part of GPU code is almost never the math. A multiply finishes in a sliver of a nanosecond. The slow part is going to fetch the numbers in the first place. Pulling a value from the GPU's main memory can cost hundreds of cycles, so a core that does a quick calculation and then waits hundreds of cycles for its next number is mostly just sitting there. The whole art of writing fast GPU code is fighting that data movement, and the single most powerful weapon for it is a tiny patch of memory called the scratchpad, or shared memory.

Start with why it is so much faster. It comes down to one trade that runs through all of hardware: distance and density against speed. The GPU's main memory is huge, tens of gigabytes, because it's built from dense, cheap cells packed tightly and sitting physically far from the cores. All that distance and density is exactly what makes it slow.

Shared memory is the opposite bet. It's built from fast, power-hungry cells sitting right next to the cores, just millimeters away on the silicon. That closeness makes it dramatically faster, on the order of a hundred times, but it also means you only get a tiny amount of it, a few tens of kilobytes per processing unit instead of gigabytes. You buy speed with area, and you can only afford a little.

Control Over the Caches

Now the part that makes shared memory special, beyond just being fast. The GPU already has automatic caches that try to keep useful data close. But those are hardware-managed, guessing on your behalf. Shared memory is the opposite; you control it. You decide exactly which data to stage there and when. That control is the whole point, because it lets you turn a pile of slow, repeated trips to main memory into one trip plus many cheap local reuses.

Here is the pattern that exploits it, and almost every fast GPU kernel uses some version of it. Imagine you need the same value from main memory not once but many times, say for a long string of multiplications, as happens constantly in matrix work. The naive way is to fetch it from far-off memory every single time, paying the long trip over and over. A hundred uses means a hundred long trips, and the cores starve waiting on every one.

Tiling: The Smart Way

The smart way works in three steps:

1. First, load once. The threads cooperate, and one of them pulls the value from main memory into the shared scratchpad, paying the long trip a single time.
2. Second, a barrier. All the threads in the team wait until the scratchpad is fully loaded before anyone reads from it.
3. Third, reuse fast. Now every thread reads that value from the scratchpad as many times as it needs, each read costing a handful of cycles instead of hundreds.

↳ You paid the long trip once and amortized it across all the reuses. This is called tiling, and it's the trick that turns a memory-bound kernel into a compute-bound one, which is exactly what you want.

The Barrier and Correctness

That middle step, the barrier, deserves a closer look, because it raises an obvious worry. What if one thread runs ahead and tries to read from the scratchpad before the others have finished filling it? Wouldn't it grab half-loaded garbage and compute a wrong answer?

It would, and that is precisely what the barrier prevents. The barrier is a rule that says every thread in the team must arrive at this line before any thread is allowed past it. So the fast threads that finish loading early are forced to stop and wait until the slowest one has placed its piece into the scratchpad. Only when the whole tile is guaranteed complete does anyone start reading. The hardware stalls the eager threads on purpose, trading a tiny wait for correctness. Without it, tiling would silently corrupt your results.

The Two Walls

Now the question that always comes up once people see how fast this is: If the scratchpad is so much quicker, why not just put the entire problem in it and skip main memory altogether?

Two walls stop you. The first is the same physics from before. The scratchpad is built from those fast, bulky cells, and main memory from dense, cheap ones. You simply cannot make the scratchpad gigabytes large without it blowing up the chip area and power, leaving no room for the cores. So it stays small by nature, a few tens of kilobytes, and most problems are far bigger than that.

The second wall is subtler and trips up even experienced people, and it is the idea of occupancy. The scratchpad is a fixed pool shared across all the teams running on a processing unit. The more of it each team grabs, the fewer teams can fit at once. Say a unit has a certain amount of scratchpad. If each team uses a modest slice, several teams fit together, and the unit has many groups of threads to juggle. That matters enormously, because the way a GPU hides slow memory is by switching to another ready group whenever one is stuck waiting. Lots of resident teams means there is always someone ready to run.

Now make one team greedy, demanding almost the entire scratchpad. Now only one team fits on the unit. And the moment that single team stalls waiting on main memory, there is no other team to switch to. The cores go idle, because there is no backup work. So you took the fastest memory on the chip, used too much of it, and ended up slower, because you starved the unit of the parallelism it needs to hide delays. This is the counterintuitive heart of it: using more of the fast memory can make the whole thing slower.