Your ALUs are idle. Memory is the real bottleneck.

*worked out from first principles*

Your GPU is rated for a staggering number of operations per second. You run a large language model on it to generate text, and the compute units sit almost completely idle, busy maybe a tenth of the time. The expensive part of the chip you paid a fortune for is dark, unused, starved.

↳ This is not a bug, not a bad driver, not lazy code. It is the central tension of modern AI hardware, and it is getting worse every single generation. Understanding why is the difference between buying hardware that helps you and buying expensive silicon that mostly waits.

The Two Ceilings

Start with the two ceilings every piece of GPU work lives under. One ceiling is how fast the chip can do math. The other is how fast it can deliver data from memory to the math units.

Which ceiling you hit depends on a single ratio: how much math you do for every byte you fetch. If your work does a lot of math per byte, you bump against the compute ceiling, and the chip is well used. If it does little math per byte, you hit the memory ceiling first, and no amount of extra compute power helps, because the math units finish instantly and then sit waiting for the next byte to arrive.

This is the whole game. Your performance is decided not by how powerful the chip is, but by whether data can reach the cores fast enough to keep them fed.

The Widening Gap

Now the trend that makes this urgent, and it is the heart of the matter. Each new generation of GPU grows its compute power dramatically, multiplying the number of math operations it can do. But memory bandwidth, how fast data moves from memory to the cores, grows far more slowly.

So generation over generation, compute races ahead while bandwidth crawls. The gap between them widens. And here is the consequence people miss: that widening gap means each new chip ships with proportionally more compute that a normal workload cannot use, because the memory cannot feed it fast enough.

The advertised operations-per-second number keeps climbing, but for a memory-bound job, more of that number is unreachable than ever before. The rating becomes a measure of theoretical capacity, not of what you will actually get.

Physics Is Not Negotiating

So why does bandwidth refuse to keep up? Why not just build memory as fast as the compute? Because bandwidth runs into hard physical walls.

It is limited by how many physical wires you can pack to carry data, by how fast you can push signals down them without corruption, by the heat and power all that movement generates. Compute grows by clever tricks like packing more math units and using lower-precision numbers, which sidestep those walls. Bandwidth has no such escape. It is fighting physics directly, and physics is not negotiating. So the imbalance is structural, baked into what is physically possible, not a temporary engineering lag.

The LLM Penalty

Now the place this bites hardest, and it explains why your language model felt so slow. Generating text with a large model happens one word at a time, and each word requires reading the model's entire set of parameters out of memory to compute that single step.

Picture that. Billions of values, hauled from memory across that limited bandwidth, just to produce one word, then hauled again for the next word, and again. The math done on each value is tiny, a quick multiply. So this is the extreme memory-bound case, almost no math per byte, and the cores spend nearly all their time waiting for the flood of parameters to arrive. You bought a machine that can do enormous math, and you are using it as an expensive device for reading memory one word at a time.

Batching to the Rescue

So how do you rescue those idle cores? And this is the key lever. The problem with generating for a single user is that you read all those parameters just to serve that one user's one word.

But if you batch many users together and generate a word for all of them at once, you read each parameter from memory a single time and immediately use it for every user in the batch. Suddenly you are doing much more math per byte fetched, because the same expensive data delivery now feeds many computations instead of one. The batch pushes your work up off the memory ceiling toward the compute ceiling, and the cores light up. This is why serving systems fight so hard to batch requests together; it is the main way to turn a memory-starved workload into a compute-rich one.

The Free Lunch Illusion

But of course the fix is not free, and the objection is real. If batching more is better, why not batch enormously? Two things break.

First, holding many users' work in progress consumes a lot of memory, and pile on enough and you simply run out, crashing. Second, the very first response gets slower, because the machine is now busy chewing through a big batch before it can return anyone's first word. So you are trading one scarce resource, bandwidth, for another, memory capacity, and trading raw speed against responsiveness. There is no free lunch, only a balance you tune.

The Split Personality of LLMs

Worth naming one more subtlety, because it shows the problem is not uniform. The same model has two very different phases.

Digesting a user's initial prompt is compute-heavy, because you process all the prompt's words together in parallel, plenty of math per byte, the cores stay busy. But then generating the reply, word by word, flips to memory-bound, one parameter-sweep per word. So a single request lives on both sides of the divide: compute-rich while reading your question, memory-starved while writing its answer. Advanced systems even split these two phases onto different machines, matching each phase to hardware suited for its bottleneck.