A GPU core can do math faster than almost anything humans have built. But it can only work on data that is physically close to it. The further data sits from the core, the longer the core waits. So chips solve this with a staircase of memory: fastest and smallest at the top, slowest and biggest at the bottom. Understanding that staircase is why some code flies and some code crawls.
↳ Think of a bank cashier. The cashier's hands and brain are the core, and the cash is the data. Where that cash is kept, and how far the cashier has to walk to reach it, is the entire story.
At the very top of the staircase sit the registers. This is the cash literally in the cashier's hand right now, the note being counted out to the customer this exact second. It is the fastest memory that exists on the chip, but it is tiny, holding only a few values at a time.
When you write a normal variable in code, like setting x to 5, that value lives in a register. It is instant to reach, but there is almost no room.
Just below that is the L1 cache (where the L stands for level). This is the cashier's own drawer right at the counter, the cash they reach for constantly. It is small, measured in kilobytes, but blindingly fast and sitting right inside the core.
Data the core keeps touching again and again gets held here so it never has to walk away to fetch it. Picture small items you need every minute kept in your shirt pocket. That is L1.
Alongside L1 there is also shared memory, which sits at the same closeness but plays a different role. It is a small tray on the counter that the cashier and the cashiers beside them can all dip into. It is how cores sitting near each other swap numbers quickly without going far.
In heavy math like matrix multiplication, hundreds of cores running together use this shared tray to hand each other values fast. Without it, that work would crawl.
Go one step down and you reach the L2 cache. This is bigger, measured in megabytes, but it sits outside the individual core and is shared by all the cores on the chip.
Because everyone has to come to this one place, it is a little slower to reach than your own drawer. Picture the bigger items that do not fit in your pocket going into a desk drawer you still reach from your chair. In the bank, this is like one larger safe in the manager's cabin that every cashier walks to when their own drawer runs dry.
At the very bottom is global memory, the VRAM. This is the bank's main vault underground, holding everything. It is enormous, holding the entire dataset, but reaching it is slow.
The cashier has to get up, walk down, clear security, and wait. This long walk down to the vault is the memory wall from the ground up. All the data is here, but pulling from here makes the core wait the longest.
Registers (Bytes) → L1 & Shared (KBs) → L2 (MBs) → Global VRAM (GBs)
Now here is what fast code actually means: a foolish cashier runs down to the vault for every customer, and the line stretches out the door. A smart cashier walks down once... stuffs it into the drawer and the counter tray, and serves everyone instantly from arm's reach.
Fast code is exactly that: keep the heavy data you touch repeatedly up near the core, in registers and cache, and walk down to global memory as rarely as you possibly can.
This is not theory, it is where real performance comes from:
And a programmer steers this on purpose. In CUDA, marking something as shared tells the chip to place that data in the fast shared tray so nearby cores can hammer it together. A plain variable drops straight into a register.
The entire craft of writing fast GPU code comes down to one obsession: dragging the data that gets reused out of the slow vault and pinning it near the core before the work starts.
The core was never the slow part...
the distance to the data is. Arrange your data so the workers rarely walk.