Two hard ceilings. One tells you to fix your math. The other says: fix your memory.

*worked out from first principles*

When a language model writes you a paragraph, it generates it one word at a time, and each new word has to look back at every word that came before it. So a naive implementation has an ugly property.

↳ To produce the tenth word it reconsiders nine words, to produce the hundredth it reconsiders ninety-nine, and by the thousandth word it is redoing an enormous pile of work it already did on every previous step. The cost of writing a passage grows with the square of its length, and almost all of that work is repetition.

The fix, called the KV cache, is what makes chatbots fast enough to exist. But the fix creates a new monster, one that quietly devours your memory and becomes the real limit on how these models are served.

The Repetition Waste

Start with the waste, because the cache only makes sense once you feel the pain it's removing. To decide how much attention a new word pays to the earlier words, the model computes two things from each earlier word, call them a key and a value, a kind of summary the new word searches against and pulls information from.

Here is the crucial fact: the key and value for a given word never change. The summary of word five is the same whether you are generating word six or word six hundred.

And yet the naive approach recomputes the keys and values for every earlier word, from scratch, every single time it generates a new word. It is like rereading the entire book from page one every time you want to write the next sentence. The information didn't change, but you paid to regenerate it anyway, over and over.

The Embarrassingly Obvious Fix

So the fix is almost embarrassingly obvious once you see the waste. Why recompute something that never changes? Compute each word's key and value once, the moment that word first appears, and stash them in a memory bank.

Then when you generate the next word, you don't recompute anything for the past, you just read the stored keys and values straight from the bank and add the one new word's entry to it. Generating each word goes from redoing all the past work to doing a fixed small amount of new work plus a lookup. The quadratic explosion collapses into something that scales gently. This stored bank of keys and values is the KV cache, and it is the single trick that turns a model that would crawl into one that responds quickly.

The Memory Tradeoff

Now the question that reveals the deeper tradeoff. This sounds like a pure win, so where is the catch? The catch is that you are making the classic bargain that runs through all of computing: you traded recomputation for storage.

You are no longer wasting compute redoing the past, but now you have to hold all those keys and values in memory for every word in the conversation, and that memory bank grows relentlessly. Every new word adds another entry that must be kept for the rest of the conversation. The longer the conversation, the bigger the cache. And it is not small, because the model stores a key and value at every one of its many internal layers, for every word. So this cache balloons into one of the largest consumers of the GPU's precious memory, often dwarfing even the model itself for long conversations.

The Bandwidth Burden

And this is where it ties back to the deepest problem of running these models. Remember that generating text is already starved for memory bandwidth, because each word requires sweeping the model's parameters out of memory.

The KV cache makes this worse in a specific way, because now, on top of reading the parameters, every new word must also read the entire growing cache of all previous words' keys and values. The longer the conversation gets, the more cache there is to haul out of memory for each single word. So the cache that saved you from recomputation now becomes a memory-bandwidth burden of its own, and eventually the dominant one. You escaped quadratic compute and landed in a growing memory problem instead.

The Capacity Wall

Worse, the cache creates a hard capacity wall. The GPU has a fixed amount of memory. The model parameters take a big fixed chunk of it, and whatever is left over is all you have for KV caches.

Since every active conversation needs its own growing cache, the amount of memory left is what decides how many users you can serve at once and how long their conversations can run before you run out. So this cache is not just a speed detail, it directly sets the economics of serving a model: how many people fit on one GPU, how long their contexts can be. Run out of cache room and you must either cut off conversations or turn users away.

Taming the Cache

Which is why a lot of clever engineering exists purely to tame this cache. The naive way of storing it wastes a great deal of room by reserving the maximum possible space for every conversation up front, most of which sits empty.

Smarter systems store the cache in small pages allocated only as a conversation actually grows, packing many users efficiently into the same memory, much like an operating system manages memory for many programs. Other techniques shrink each entry by storing it in a compressed, lower-precision form, or let several of the model's attention streams share one set of stored keys and values instead of each keeping its own, cutting the cache size by a large factor. All of these exist for one reason: the KV cache is the scarce resource, and squeezing more conversations into limited memory is the name of the game.