Why GPUs care so much about memory bandwidth

*worked out from first principles*

A GPU core does math blindingly fast. Billions of operations a second. But most of the time it isn't doing math at all. It is sitting idle, waiting for data to show up. That one fact, more than core count, more than FLOPS, decides how fast a chip actually runs.

It is easy to say thousands of cores run at once. But they don't, not really, not unless something keeps feeding them. That something is bandwidth.

↳ So the real story isn't the cores. It's what feeds them. Let's build it from the bottom.

First, why does a core starve?

It's a speed gap, and the numbers are brutal. A GPU core finishes one piece of math in roughly 1 unit of time, but fetching fresh data from main memory takes closer to 100 or 200 units. The data lives on the far side of the chip, and getting it across takes forever in core-time. So the worker finishes, looks up, and there is nothing in his hands. He sits. That gap between a fast core and slow memory has a name: the memory wall.

And this isn't bad engineering, it's physics. Inside the chip, data travels through microscopic wires. The math inside a core happens at basically the speed of electricity, but pushing data through those thin wires has a hard physical ceiling. Only so much can flow through at once, and you can't just wish more through. That is the wall, and it is built into the silicon.

Scaling it up

Now multiply that by scale. A CPU has a few cores, so its modest appetite is easy to feed. A GPU has thousands of cores, all screaming for data at the exact same instant, all shouting "give me numbers, give me numbers."

If the pipe feeding them is narrow, a handful get fed and the other thousands starve. The more cores you add, the worse the starvation gets unless the pipe grows with them. Raw core count without bandwidth is just a factory full of idle workers.

So what's the fix? You widen the road.

Ordinary RAM feeds data through a narrow pipe, which is fine for a CPU but useless for thousands of hungry cores. So GPUs use special memory built for width, specifically HBM or GDDR.

↳ Think of it as swapping a garden hose for a firehose. Now data arrives in a flood, the workers always have something in their hands, and they actually feast instead of starve.

That flood of supply is what bandwidth really means: not how fast one delivery is, but how much total data arrives per second.

The second trick: Latency Hiding

There is a clever second trick stacked on top: latency hiding.

Fetching a specific piece of data still takes a long time. But a GPU core doesn't fold its arms and wait for it. The moment its data is delayed, it grabs a different ready piece of work and starts on that instead. By the time the slow data finally lands, the core has already done useful work on something else, meaning the delay got hidden behind other work.

↳ The catch is that this only works if the wide pipe keeps a steady flood of fresh work arriving. Starve the pipe and there is nothing to switch to, and latency hiding dies.

Which chip do you actually need?

Every workload is bottlenecked by one of two things, and they want opposite hardware:

1. Memory-bound: The math is easy but the data is enormous. Imagine copying a massive database of user profiles. There is no calculation, just endless hauling of files from memory to the processor. Here, faster compute cores buy you nothing because the math unit was never the problem. The connection road was. You need more bandwidth. Throwing FLOPS at this is wasted money.
2. Compute-bound: The data is tiny but the math is brutal. Imagine decrypting a password. The input is just a single 8-character string, but you have to run it through millions of complex mathematical equations. Here, widening the memory highway does nothing because there is barely any data to move. You need faster processors and more raw compute. Throwing bandwidth at this is wasted money.

This is why "which GPU is best" has no single answer. A job hauling oceans of data wants HBM-class bandwidth. A job grinding heavy math on little data wants raw compute power. You match the hardware to the bottleneck, not to the spec sheet.

GDDR vs. HBM: Speed vs. Width

Which brings up the next question: GPUs use HBM or GDDR, so what is the actual difference? They are both the GPU's memory, but they bet on opposite things. One bets on speed, the other on width.

GDDR is the fast lone delivery boy. It sits as separate little chips on the board around the main GPU, and it rides a relatively narrow road. But it cycles that road at a very high frequency. It is cheap, widely available, and perfect for gaming, where the data isn't huge but it needs to arrive quickly. One fast rider on a thin street.

HBM is the massive cargo train. Instead of scattering chips around the board, HBM stacks them vertically, one on top of another like a tower, and parks that tower right next to the GPU core. It runs on an absurdly wide road, thousands of microscopic wires moving data side by side. Any single lane may be slower than GDDR, but the road is so wide it dumps an entire truckload in one pass.

↳ That is why every serious AI chip (like the H100s and A100s training large models) uses HBM. LLMs need oceans of data moved in bulk with no traffic jam, and width is the only thing that delivers that.

The cost is that HBM is expensive and hard to manufacture, which is exactly why it lives in data centers and not your gaming PC.