Why a GPU Multiplies Matrices Faster

*worked out from first principles*

Last chapter, we broke down why a CPU can't have 10,000 cores. This is the other half: why a GPU can, and why it only wins at one very specific shape of work.

Here's the thing almost everyone repeats: "A GPU is faster because it has more cores." I believed it too for years. But it bugged me, because more of something slow doesn't automatically beat fewer of something fast. A bus has more wheels than a Ferrari. That doesn't make it quicker to the airport.

↳ So the claim is incomplete. The real question isn't "does the GPU have more cores." It's "what kind of work lets thousands of weak cores beat a handful of strong ones?" And the answer isn't in the chip. It's in the math.

First, what matrix multiplication actually is.

It's rows times columns, multiply and add. Take one row from the first matrix, line it up against one column from the second, multiply them pair by pair, add the results. That single number is one cell of the answer grid.

↳ Answer[slot 1] = (A × X) + (B × Y) + (C × Z)

Now sit with the part that changes everything. Every cell of the answer is independent. For a 3x3 times a 3x3 you get 9 output cells, and not one of them needs another one to be done first. Cell 1 doesn't read Cell 5. Cell 9 doesn't wait on Cell 2.

↳ There is no order. They're 9 separate little math problems that happen to share input, and they could all be solved at the exact same instant without anyone noticing. That independence is the whole secret. Now watch why the same work splits two completely different ways.

A CPU goes serial

A CPU has a few big, fast, expensive cores. So it does the only thing it can: it goes serial. Cell 1, then Cell 2, then Cell 3, one clock cycle after another.

For 9 cells that's nothing. But real AI is not 3x3. A single transformer layer inside a model like ChatGPT multiplies matrices around 10,000 × 10,000. That's 100 million independent cells in one operation. A CPU marching through 100 million cells one at a time, however fast each step is, is going to drown.

A GPU splits the work

A GPU looks at those same 100 million independent cells and asks the obvious question: why are we standing in a line at all?

So it hands each output cell to its own tiny core. One worker per slot. Millions of slots, millions of workers, all firing together. But to really get why a CPU can't just copy that trick, you have to go one level down...

Inside the core: ALU vs. Control Unit

Most people blur the inside of a core into one thing, but it's really two main parts:

1. The ALU (Arithmetic Logic Unit) is the worker. And here's the surprise: the worker is dumb. It's just a circuit of transistors. It cannot think. It has no memory of its own. It sits there and waits to be told what to do.
2. The Control Unit is the manager. It reads the code and drives the ALU. It fetches the numbers, sends an electrical signal (like 001 for add, 010 for subtract), which flips the right transistors inside the ALU to drop the answer out.

Every "smart" thing a CPU does (guessing your next instruction through branch prediction, or reshuffling work to stay busy via out-of-order execution) lives inside this Control Unit manager.

↳ Core = Manager (Control Unit) + Worker (ALU) + Pocket Memory (Registers)

The exact fork: CPU vs. GPU

This is where a CPU and a GPU stop being the same chip.

A CPU core gives one big, smart manager to just a few workers. Every single core makes its own complicated independent decisions. That manager is enormous (billions of transistors), so only a few cores physically fit.

A GPU core does the exact opposite. It takes one manager and puts it in charge of thousands of workers. One manager shouts "001" a single time, and thousands of workers all add at the exact same instant. No worker thinks for itself. That's SIMT (Single Instruction, Multiple Threads). One command, a whole army moving in lockstep.

↳ So the trade is clean. A CPU core is one genius working alone, fast and independent. A GPU replaces the genius with thousands of obedient workers who only know how to follow the same order together. On an A100 or H100, that's roughly 5,000 to 10,000 of these tiny cores on one chip.

Feeding the army

Thousands of cores are useless if data trickles in. Two things prevent the army from starving:

• Bandwidth: A GPU skips ordinary RAM and uses wide memory like HBM. An NVIDIA A100 moves around 1.5 to 2 TB/s of memory. Think of it as a 10-lane highway built only to keep workers fed.
• Arithmetic Intensity: This is the deepest point. Matrix math pulls a number in once and reuses it across many rows and columns. So much math happens per byte that the cores are buried in computation. The memory delay hides behind the massive workload (on an A100, that's the gap between ~2 TB/s memory and 300+ TFLOPS of compute).

Put all of it together and you can feel it in something you use every day. Every time you type into ChatGPT, billions of these matrix multiplications run behind your answer. On a CPU, line by line, that takes minutes and the chat hangs. On a GPU, split across thousands of cores, the same work comes back in milliseconds.

Why not run your OS on it?

If the GPU is such a monster, why don't we run Windows or Linux on it? Because the moment work stops being independent, the entire model collapses.

• Sequential code: You can't pour the milk before the tea has boiled. On a sequential task, exactly 1 of the 10,000 cores does anything, and the other 9,999 sit swatting flies.
• Low arithmetic intensity: Normal apps mostly just move data around without heavy math. Cores spend time waiting on memory, and that beautiful bandwidth advantage disappears.
• Amdahl's Law: A program runs only as fast as its slowest sequential part. One stubborn serial section puts a ceiling on the entire chip.
• Slow single core: One GPU core, on its own, has a low clock speed. In a one-on-one fight on a sequential task, a single CPU core wins easily.