Generating text from a large model is painfully slow for a reason that feels almost insulting. The model produces one word at a time, and to produce each single word it has to haul its entire enormous set of parameters out of memory.
↳ So the expensive compute cores do a tiny bit of work, then wait a long time for the next sweep of the model to arrive, then do a tiny bit more. The cores are mostly idle, starved, waiting on memory.
Speculative decoding is a trick that fills that idle time by turning generation into a game of guess and check, and it can make a model produce text several times faster while giving you the exact same words it would have produced anyway. How it pulls that off is genuinely clever.
Start with why one word at a time is so wasteful. Each time the big model runs, it drags all its parameters from memory just to produce a single word, and the actual math involved is trivial compared to the cost of that memory sweep. So the model has enormous compute capacity sitting unused on every step, because the bottleneck is the data movement, not the arithmetic.
Here is the key realization hiding in that waste. Running the big model to check five words at once costs almost the same as running it to produce one word, because either way you pay for one sweep of the parameters from memory, and the extra math to handle five words instead of one is nearly free on those idle cores.
Producing is serial and slow. Checking many at once is nearly the same price as checking one. That asymmetry is the whole opening.
So the plan. What if you had a second model, a small fast one, that could cheaply guess the next several words? This little model is far less capable, but it's also far smaller, so it can generate words quickly because its parameter sweep from memory is tiny.
You let this small draft model race ahead and blurt out a guess for, say, the next five words. These are just guesses, some right, some wrong, but they were cheap to produce.
Now the beautiful part. You take those five guessed words and hand them all to the big model at once, and ask it to check them in a single pass. And remember, checking five words costs the big model almost the same as producing one, because it's the same one memory sweep.
In that single pass, the big model effectively says, for each guessed position, whether that is the word it would have chosen itself. It works through the guesses and accepts them one by one as long as they match what it would have said, and stops at the first guess it disagrees with. Everything up to that first disagreement is confirmed correct, exactly what the big model would have produced on its own, and at the point of disagreement the big model supplies its own correct word instead.
So think about what just happened. If the little model guessed five words and the big model agreed with the first three, then in a single expensive pass—the price of normally producing just one word—you produced four correct words: the three accepted guesses plus the one correction.
You got several words for the price of one memory sweep. The small model did the cheap speculative labor, and the big model, instead of slowly birthing one word, ratified a whole batch in the time it would have taken to make one. The idle compute that was going to waste is now spent verifying guesses, which is nearly free, and it buys you multiple words per expensive step.
Now the objection that should be nagging you, because it sounds like it might cheat. If a small dumb model is doing the guessing, aren't you degrading the output? Isn't this a quality shortcut where you get worse words in exchange for speed?
No, and this is the elegant heart of it. The small model never gets the final say on anything. It only proposes. Every single word that survives is one the big model explicitly confirmed it would have chosen itself, and any guess the big model disagrees with is thrown away and replaced with the big model's own choice.
So the output is word for word identical to what the big model would have produced alone, just arrived at faster. The little model is only there to guess, the big model is the sole authority on what actually gets written. Speed went up, quality did not move a hair. That's what makes the trick honest rather than a shortcut.
Of course there is a cost, and it lives in the guessing. The whole scheme only pays off when the little model guesses well, because every accepted guess is a word you got nearly for free.
If the draft model is decent and its guesses are often right, you accept long runs and fly. But if the draft model is bad, or the text is hard to predict, most guesses get rejected, and then you wasted effort running the draft model and checking guesses that got thrown away, and you crawl back toward the original one-word-at-a-time speed, now with extra overhead on top. So the art is in choosing a draft model that's small enough to be cheap but smart enough to be right often, and the actual speedup you get depends entirely on how often its guesses are accepted. Easy, predictable text flies. Hard, surprising text barely benefits.
So the whole picture. A big model generating one word per memory sweep leaves its compute cores mostly idle, because the cost is in moving the parameters, not the math, and checking many words costs nearly the same as checking one.
Speculative decoding exploits this by letting a small fast model guess several words ahead, then having the big model verify all of them in a single pass, accepting the run of correct guesses and fixing the first wrong one. You get multiple words for the price of one expensive step, and because the big model confirms every surviving word, the output is exactly what it would have produced alone.
When your expensive resource is idle because it's waiting on something slow,
you can often fill that idle time with cheap speculative work and verify it in bulk,
getting real speed from capacity that was otherwise going to waste.