Why CPUs Can't Have 10,000 Cores

*worked out from first principles*

I used to wonder why a CPU only packs 8 or 12 cores while a GPU easily fits thousands. They use the exact same silicon and obey the same physics. So where does the massive gap come from? Well, it turns out chipmakers can actually put 10,000 tiny compute units on a single chip. They already do it every day. We just call that chip a GPU.

↳ The real question I needed to ask wasn't "Why can't a CPU have 10,000 cores?" It was more like... "Why is a single CPU core so incredibly expensive to build that we can only squeeze a few dozen onto a die?" To figure that out, I had to break it down from first principles.

First, what is a chip actually made of?

Let's start at the very bottom. The core isn't the smallest unit. The transistor is. Think of a transistor as a microscopic light switch:

• On means current flows, which the computer reads as a 1.
• Off means current stops, reading as a 0.

That is literally it. The whole machine only understands those 1s and 0s, and transistors are just the physical things flipping between them.

One switch on its own is basically useless. But if you wire billions of them together, they start communicating. One switch tells the next one, "hey, if I'm on, you turn off." This massive web of switches is how a chip calculates math, renders video games, and makes decisions. They are so ridiculously tiny that you could fit thousands of them across the width of a human hair.

So the ladder looks like this:

1. Transistor = a single brick. Just an on/off switch.
2. Core = a whole room. Billions of transistors wired together into a brain. Funny enough, this brain mostly guesses things instead of doing actual math.
3. CPU = the entire building. A few dozen of these rooms sitting under one roof.

Why not shrink them and cram thousands onto the board?

It all comes down to what actually goes INSIDE a core to make it fast. This is where the big split happens. A CPU is built for latency. It wants to finish one specific task as fast as humanly possible. A GPU is built for throughput. It wants to finish thousands of slower tasks all at the same time. Hold onto that idea because everything else makes sense once you get that.

1. A CPU core's entire job is chaotic and unpredictable. One millisecond you are gaming, the next you open Chrome, then a background app updates. The instructions change constantly. So the core has to be "smart", and being smart costs a ton of physical silicon.
2. Most of that silicon is actually spent guessing instead of doing math. Branch prediction tries to guess your next instruction before it even runs. Out-of-order execution scrambles instructions on the fly to keep the core busy. These are massive guessing engines that take up physical space to keep the math units fed.
3. Then you have cache. Standard RAM is way too slow to feed a fast core. So CPUs put huge memory pools directly on the chip like L1, L2, and L3 cache. On a real CPU die, cache can take up more than half the total chip area! The actual math units just sit in a tiny corner.

Because of all this, a CPU core is massive. Billions of transistors get eaten up by predictors and private caches. You physically hit a ceiling at a few dozen cores.

A GPU core, on the other hand, is just a tiny, dumb math unit. No huge cache. No guessing hardware. If you strip away all that "smart" stuff, a core shrinks by a factor of 100. That is exactly how you fit thousands of them.

Latency vs Throughput

The core concept underneath all of this architecture is Latency vs Throughput.

• Latency (CPU) = time to finish one task from start to finish.
• Throughput (GPU) = total work done over a window of time.

A CPU chases zero delay for a single heavy task. A GPU chases thousands of smaller tasks finishing together. It is like comparing one insanely fast sports car dodging traffic versus a massive fleet of slow cargo trucks.

So why not run Windows on a GPU?

You might wonder why we don't just run an operating system on a GPU to make it crazy fast. Well, an OS is exactly the type of workload a GPU is terrible at.

• Endless branching: An OS is constantly making if/else decisions based on whatever the user decides to click.
• Lockstep requirement: GPUs demand all their thousands of cores do the exact same step at the exact same instant.
• Thread divergence: If those cores hit different branches in the code, they experience divergence. They can't run together anymore and have to take turns. All that amazing parallel speed collapses into a slow serial crawl.
• Hardware interrupts: An OS has to react instantly to random hardware events like a mouse movement. CPUs handle those seamlessly. GPUs hate stopping their huge workflows for sudden interruptions.

↳ The chaotic code that a CPU eats for breakfast is the exact thing that breaks a GPU's lockstep model.