What if a GPU had CPU-style caches

*worked out from first principles*

Here's a tempting idea. Cache is the fastest memory on a chip, the stuff that keeps data right next to the cores so they never wait.

A CPU has lots of it. A GPU has very little. So why not just give a GPU a big fat CPU-style cache and make it even faster. It sounds obviously good. It is actually a disaster, and walking through exactly why it fails teaches you more about chip design than almost anything else.

The Zero-Sum Game of Silicon

The whole thing rests on one hard fact. A chip is a fixed size.

That limit isn't a budget you can argue with, it's physics. The chip stays roughly the size of a biscuit. So every square millimetre of silicon is a zero-sum game. Whatever you spend on one thing, you take away from something else.

You cannot grow the cache and keep all the cores. The cache has to come from somewhere, and that somewhere is the workers.

The Cache Slider (Thought Experiment)

Picture it as a single slider you can drag, splitting the chip between cache and math cores.

Slide it all the way toward cores and you get something like 85 cores and a tiny sliver of cache. That's GPU territory, a small cache so that thousands of cheap workers fit, all grinding away. It's built for throughput, doing millions of simple things at once. This is what a GPU is.

Now start dragging the slider toward cache. Push it to where cache takes about a third of the chip, and your core count drops, maybe from 85 down to 70. And now you have the worst of everything. It's not a proper CPU, because it doesn't have the deep cache and clever single-thread machinery to finish one task blazingly fast. And it's not a proper GPU, because you've thrown away a chunk of its army. It's a jack of all trades and master of none, an expensive design that lost both superpowers at once. This hybrid middle is a trap.

Drag further, to where cache eats about two-thirds of the chip, and now you're in CPU territory. Lots of cache, big and close, so a single task almost never waits. Very low latency. But the core count has collapsed, maybe to 35, and it is now genuinely bad at parallel work. It can sprint a single task but it cannot field the crowd that AI and graphics demand.

And just to see the logic end-to-end, slide it all the way to 100 percent cache. Now the entire chip is storage and there are zero cores. You have built a beautiful warehouse with not a single worker in it. Production is zero. It does no math at all. That's the absurd endpoint that proves the point: more cache always means fewer workers, right down to none.

The Sneaky Cost: Cache Coherency

So that's the first cost, you trade away the workers that are the entire reason a GPU exists. But there's a second, sneakier cost that makes big caches on a GPU even worse.

It's called cache coherency, and it burns power for nothing.

Imagine a factory where every worker keeps their own little notebook, their own private cache. Now one worker changes a number, say updates a value from 5 to 10. The problem is every other worker who has that number in their notebook is now holding a stale, wrong value. So the system has to race around telling all of them, the number changed, fix your notebook.

With a handful of CPU cores that's manageable. But a GPU has thousands of workers. Keeping thousands of private caches all agreeing with each other means a constant storm of these update messages flying across the chip. That bookkeeping eats enormous power and does zero actual math.

So a GPU with thousands of cores and big private caches would spend its energy mostly on keeping everyone's notebooks in sync, not on computing. The very thing that makes cache helpful on a few cores makes it a power-burning nightmare on thousands.