For years, a GPU's cores did math one small operation at a time: one multiply, one add, over and over. Then the workloads shifted to AI, which is almost entirely one operation: multiplying matrices, done at colossal scale.
↳ So chipmakers did something drastic. They carved out a piece of the chip and built a circuit that does nothing but multiply small matrices, in a single shot, dozens of times faster than the regular cores could.
These are tensor cores, and they are the reason a modern GPU can train models that would have been unthinkable a decade ago. Understanding them shows you what happens when hardware stops being general and becomes a specialist.
Start with how a regular core works, because the contrast is the whole point. An ordinary GPU core is a generalist. It can add, multiply, compare, do all sorts of operations, but only one at a time, in sequence.
The fundamental step in multiplying matrices is to take two numbers, multiply them, and add the result to a running total: a multiply-then-add. A regular core does one of these per tick. And multiplying two large matrices requires an astronomical number of these multiply-adds. So even with thousands of regular cores each grinding through them one per tick, the sheer count of operations is enormous, and that is the ceiling on how fast the math can go.
Now the question that led to tensor cores: if the entire workload is dominated by this one pattern, multiply-add repeated billions of times in the specific structure of a matrix multiply, why keep doing it one step at a time on general hardware? Why not build a dedicated circuit that swallows a whole chunk of that pattern in one bite?
That is exactly what a tensor core is. Instead of doing one multiply-add per tick, a tensor core takes two small matrices as input and computes their entire product in effectively a single operation.
Picture a small grid of tiny multiply-add units all wired together in hardware, arranged so that the numbers flow through them and every multiply-add needed for that little matrix multiply happens at once, in parallel, in one coordinated step. What would have been many separate instructions on a regular core, marching through row-times-column one pair at a time, collapses into a single hardware action that produces the whole small result grid together.
So where a regular core does one multiply-add, a tensor core does a whole batch of them fused into one step, which is where the huge speedup comes from.
And this raises the obvious question: if this specialized circuit is so much faster, why not build the whole chip out of tensor cores and forget the regular ones?
Because a tensor core is a rigid specialist. It does exactly one thing, multiply small matrices, and nothing else. It cannot run your general logic, cannot handle branches, cannot do the thousand other operations a real program needs. The moment your work is not a matrix multiply, the tensor core is useless.
So a GPU keeps both: a sea of flexible general cores for all the varied work, and a set of tensor cores for the one specialized pattern that dominates AI. It is the classic tradeoff: specialized hardware is dramatically faster at its one job and worthless at everything else, so you include it alongside the generalists rather than replacing them.
There is a second key part of how tensor cores get their speed, and it ties into a theme that runs through modern AI hardware. They typically take their inputs in a lower-precision form, numbers stored with fewer bits, coarser and less exact.
Feeding smaller, coarser numbers lets the circuit pack in more multiply-add units and run them faster, because each one is doing cheaper arithmetic. And this works for the same reason quantization works: neural networks are tolerant of imprecision, so doing the multiplies in a coarse form barely affects the final result while dramatically boosting speed.
To keep things from drifting too far, the circuit often accumulates its running totals in a more precise form even while multiplying in the coarse one, getting most of the speed of low precision without the full loss of accuracy. So tensor cores are fast partly because they are specialized, and partly because they lean into low precision exactly where the math can tolerate it.
Now it is worth connecting this to why you still need everything else you know about the memory system. A tensor core is so fast at the math that it makes the imbalance between compute and memory even more extreme.
The circuit can consume small matrices and spit out products at a blistering rate, which means it is even hungrier for data than a regular core. So feeding it becomes the whole challenge, and all the techniques for staging data in fast memory and reusing it heavily become more important, not less. A tensor core starved for data is an even bigger waste than a regular core starved for data, because it could have done so much more. The specialist circuit made the math so cheap that moving the data is now almost the entire game.
So the whole picture. Regular GPU cores are generalists that do one multiply-add at a time, and matrix multiplication needs an astronomical number of those, which caps how fast AI math can run. Tensor cores are specialized circuits that compute a whole small matrix product in a single fused hardware step, dozens of times faster, by wiring many multiply-add units together and running them at once.
They cannot do anything but matrix multiply, so they live alongside the flexible general cores rather than replacing them, and they lean on low precision where the math can tolerate it for extra speed. And because they make the math so blindingly fast, they push the real bottleneck even harder onto the memory system, making the art of feeding them the central problem.
The deep lesson is that when one pattern dominates everything,
the winning move is to burn silicon on a circuit that does only that pattern,
and then spend all your remaining effort just keeping that circuit fed.