Why not build a GPU entirely from registers?

*the art of being approximately right*

A large language model is, underneath everything, a gigantic pile of numbers. Billions of them, the learned weights that encode what the model knows. And normally each of those numbers is stored in a fairly precise form, taking a certain chunk of memory each.

↳ Quantization is the trick of storing each number in a cruder, smaller form, using far fewer bits, and it is one of the most important techniques in making these models runnable at all.

It can shrink a model to a fraction of its size and make it several times faster, and the surprising part is that the model still works almost as well. Understanding why you can be so sloppy with the numbers and get away with it teaches you something deep about both neural networks and the hardware they run on.

The Target: Moving Bytes

Start with why you would want to be cruder on purpose. Everything about running these models keeps coming back to one bottleneck: moving data. The weights have to be hauled out of memory to do any computation, and there are billions of them, so the sheer volume of bytes moved dominates everything.

Now suppose each number, instead of taking a full precise slot, took a quarter of that. Instantly the whole model takes a quarter of the memory to store, and more importantly, a quarter of the bytes to move for every operation. Since moving bytes was the thing choking performance, cutting the bytes by three quarters is a massive win, both in speed and in how much model you can fit on a given chip. Quantization attacks the exact resource that everything else is starved for.

Why the Model Doesn't Break

So the obvious objection, and it's a good one. These numbers were presumably precise for a reason. If you chop them down to a crude approximation, surely you wreck the model. Round off billions of numbers and the errors should pile up into garbage.

Here is why they don't, and it's genuinely surprising. A neural network is not a delicate calculation where a tiny error in one place ruins the answer. It is a vast, redundant web where the final result comes from combining an enormous number of these values, and the meaning is spread across many of them rather than resting on any single precise one.

So when you round each weight a little, the errors are small, they point in random directions, and they largely cancel out when combined across billions of terms. The network was trained to be robust to noise in the first place, so a bit of extra rounding noise is just something it shrugs off. It is like a photograph made of millions of pixels: you can reduce the number of distinct colors each pixel uses quite aggressively and the picture still looks fine, because the image lives in the whole, not in the exact shade of any one dot.

How to Crush a Number

So how do you actually crush a number into fewer bits? The core idea is that you rarely need the full range that a precise number can represent. A given batch of weights might all live in some narrow band of values.

So instead of storing each one in an expensive general form, you find the range that batch actually spans, and you carve that range into a small number of evenly spaced levels, and store each weight as just which level it is closest to. With only a handful of bits you get a modest number of levels, and you snap each true value to the nearest one. You also remember the scale, the mapping from levels back to real values, so you can approximately reconstruct the originals when you need to compute. Crude, but for a robust network, close enough.

The Limits and the Outliers

Now the question that separates a toy understanding from a real one. If fewer bits is cheaper, why not go absurdly low, why not one or two bits per number?

Because two things break as you get cruder. The first is that with too few levels, the gaps between them get wide, and the rounding error on each number grows large enough that it stops being harmless noise and starts genuinely distorting what the model knows. There is a floor below which the approximation is too coarse and quality falls off a cliff.

The second problem is subtler and is the real villain: outliers. Most of the weights huddle in a narrow range, but a few are dramatically larger. And if you stretch your small set of levels to cover those rare giant values, then all the normal values get crammed into just a couple of levels near zero, losing almost all their distinction. One extreme value ruins the resolution for everyone else. Handling these outliers gracefully, by giving them special treatment or storing different parts of the model at different crudeness, is most of what makes modern quantization actually work rather than fall apart.