Splitting the giant

*Why "local" memory is not local at all*

Modern models have grown so large that they no longer fit on a single GPU. Not the model's parameters, not the memory it needs to run, none of it. So you are forced to spread one model across many GPUs working together.

↳ And this sounds simple, just cut it into pieces, but it opens up the hardest problem in large-scale AI. Because the moment a model lives on many chips, those chips have to constantly talk to each other, and that talking becomes the new bottleneck.

There are three fundamentally different ways to split a model across GPUs, each with its own pattern of communication and its own way of hurting. Understanding them is understanding how models the size of the ones you use every day are actually run.

The Unavoidable Need to Talk

Start with the reason you are forced into this at all. A GPU has a fixed, limited amount of memory. A large model's parameters alone can be far more than that, and running it needs even more memory on top. So the model physically cannot sit on one chip.

You have no choice but to divide it across several, and the only question is how you cut it. And every way of cutting creates a need for the GPUs to exchange data, because a model is one connected computation, and if you split that computation across chips, the pieces have to pass results between them. The entire art is choosing cuts that minimize how much the chips must talk.

1. Data Parallelism

The first way is the simplest, called data parallelism, and it applies when the model does fit on one GPU but you want to train it faster on more data. You put a complete copy of the model on each GPU, and feed each copy a different slice of the data.

Each GPU computes independently on its own slice, which is wonderfully parallel with no talking during the work. But there is a catch at the end. Since they are all training one shared model, after each step they must combine what they each learned, averaging their updates together so all the copies stay identical. That averaging step requires every GPU to share its results with all the others, a burst of heavy communication. So data parallelism is nearly free during the computation and then pays a large communication bill at the synchronization point. And it does not solve the core problem, because it requires the whole model to fit on each GPU in the first place.

2. Tensor Parallelism

So what happens when the model itself is too big to fit on one chip? Now you have to split the model, not just the data.

The second way, tensor parallelism, slices individual layers of the model across GPUs. Take one big mathematical operation in the model and cut it lengthwise, so each GPU holds a fraction of that layer and does a fraction of the work.

The problem is immediate and severe. Because you split a single operation across chips, the chips must combine their partial results to complete that one operation, and this happens deep inside the model, for every layer, on every single token. So tensor parallelism generates constant, intense communication, chips exchanging data many times per token as the computation flows through the split layers. This only works when the GPUs are connected by an extremely fast, short link, because the talking is so frequent that any delay would dominate everything. You keep tensor parallelism inside a tightly connected group of chips sitting right next to each other, never spread across a slow network.

3. Pipeline Parallelism

The third way, pipeline parallelism, cuts the model the other direction. Instead of splitting each layer, you put different whole layers on different GPUs: the early layers on the first chip, the middle layers on the next, the late layers on another, like stations on an assembly line.

Data flows through, chip one does its layers then passes the result to chip two, which does its layers and passes to chip three. The communication here is light, just handing off a result once between neighboring stages, not the constant chatter of tensor parallelism.

But it has its own ugly problem. When the first chip is working on the first stage, all the other chips are sitting idle waiting for work to reach them, and when the work has moved down the line, the early chips are now idle. This idle time is called the bubble, and it is pure waste, GPUs doing nothing while they wait their turn in the line.

So here is the question that reveals the real engineering: If pipeline parallelism leaves chips idle waiting their turn, how do you keep them all busy?

The answer is to not send one piece of work down the line at a time, but to keep a steady stream of them flowing. You break your batch into many small pieces and feed them into the pipeline one after another, so that while chip two is working on the first piece, chip one is already working on the second piece, and soon every chip is busy on a different piece at once, just like a real factory line running full. You cannot eliminate the bubble at the very start and end when the line is filling and draining, but you can shrink it to a small fraction by keeping the pipeline packed with work.

Braiding them together

Now the truth about how the biggest models actually run, which is that you don't choose one of these, you combine all three at once.

You use tensor parallelism inside a small group of tightly linked chips to split each layer, because they can talk fast enough to afford the constant chatter. You use pipeline parallelism to spread the model's layers across several such groups, handing off between them with light communication. And you wrap the whole thing in data parallelism, running many complete copies of this multi-GPU model on different data slices to train faster. So a single enormous training run might spread across thousands of GPUs, organized in this layered way, each type of split chosen to match how fast the underlying connections are.