Here's the interview question that separates people who memorized facts from people who actually understand hardware. Design a chip to train a model like ChatGPT.
If your answer is just add more cores, you fail on the spot. Because the deepest truth about training an AI model is this: the math is cheap. Moving the data around is what costs everything. Once you believe that, the entire design follows, and it pulls together every idea about how these chips work into one blueprint. Let's build it from scratch.
The chip rests on five decisions. Each one is a choice about what to put on the silicon and, just as importantly, what to throw off it. Remember the chip is a fixed size, so every choice is a trade: you only get to add something by removing something else.
First, the compute. What do the workers look like?
Training is heavy, repetitive, predictable math, the same matrix multiplications over and over. So you do not want clever CPU-style cores with their big caches and guessing machinery. You rip all of that out. You fill almost the whole chip with simple, hardwired units built for one job, multiplying grids of numbers as fast as possible. Cheap workers in huge numbers, not a few expensive geniuses.
Now a sharp objection. If you strip out the big caches, doesn't the data have to come from far-off memory every time, starving those workers?
And the answer is the quiet reason this works. In training, once a chunk of data arrives at the cores, they reuse it an enormous number of times before needing anything new. The data comes in once and gets ground on for ages. A browser or a database jumps around memory randomly and genuinely needs a big cache, but this workload doesn't. So the cache would mostly sit wasted, and that silicon is far better spent on more math units.
Second, feeding those workers. With the whole chip now packed with hungry math units, ordinary memory would leave them starving, idle most of the time waiting on data that trickles in.
So you bring the memory physically onto the package, stack it into a tower right beside the compute, and connect it with thousands of microscopic vertical wires instead of a few long ones on a board. The supply road becomes absurdly wide, and the data arrives as a flood instead of a trickle. The workers feast instead of starve. This is the single most important decision, because without it everything else is pointless.
Third, managing all those workers. You have thousands of them, and if each one needed its own manager, the management hardware would eat the chip and melt under its own bookkeeping.
So you bundle the workers into fixed groups and put one small shared manager over each group. The whole group takes one instruction and executes it together in lockstep.
And the beautiful part is that AI training fits this perfectly, because there's no branching in it. Even the part where the model corrects its own mistakes, the backward pass, is not a maze of if/else decisions. It's just matrix multiplication run in reverse, more of the same predictable math. So the workers never split off down different paths, the groups stay locked together, and that tiny shared manager is all you ever need.
Fourth, the delays. Even with that flood of memory, data still takes time to arrive, and the workers will sometimes have to wait. You do not solve this by making the workers smart enough to predict, that would cost the silicon you just saved.
Instead you keep a huge pile of ready work loaded up, and the moment one group stalls waiting on data, the scheduler instantly switches to another group that has work ready. By the time it cycles back, the first group's data has landed. The math units never sit idle, they just keep changing which group they're serving.
And there's a subtle trick that makes this switching free. On a CPU, swapping tasks means saving the old task's state out to memory and loading the new one, which costs real time. This chip avoids that entirely. It keeps the state of every waiting group already sitting in fast on-chip registers, all the time. So when one group stalls, nothing has to be saved or loaded. The scheduler just flips a switch to the next group's registers, and the swap happens with essentially zero delay. That is what lets the switching be aggressive enough to hide the waiting completely.
Fifth, and this is the one people forget: scale. A real model is far too big to fit on a single chip. It has to be split across thousands of them.
And the instant your chips have to talk to each other over the slow standard slots a normal computer uses, the whole system chokes, your expensive chips sit idle waiting to pass data back and forth. It's like joining two bullet trains with a bullock-cart track between them.
So you build dedicated high-speed links directly between the chips that bypass the slow general-purpose routes entirely, fast enough that thousands of chips behave like one single giant brain rather than a crowd shouting across a room. For a model this size, this communication is the ultimate bottleneck, and getting it wrong wastes everything else you built.
So step back and look at the whole blueprint. You threw out caches and filled the space with math units. You brought memory in close and made the road to it enormous. You bundled the workers under tiny shared managers because the work never branches. You hid the unavoidable delays by always having other work ready and keeping its state on-chip so switching is free. And you wired thousands of chips together with direct high-speed links so they act as one.
Every single one of those choices is the same instinct repeated. Spend silicon and effort on moving data and keeping the workers fed, not on making any individual worker clever.
Because in training a model, the math was never the hard part.
Keeping the data flowing was the whole game.