Why branching is bad on GPUs

*worked out from first principles*

A GPU's whole speed trick rests on one rule. A group of 32 threads moves together, takes one order at a time, and does the exact same thing in lockstep. That works beautifully until the code asks a question: an if/else.

The moment a warp hits a decision that splits its threads down two different paths, that speed advantage doesn't just shrink. It gets cut in half. This is warp divergence, and it is the single most expensive mistake in GPU code.

The classroom analogy: A split path

Start with the picture that makes it obvious. Think of a classroom with one teacher and 32 students. The teacher is the control unit, the single brain. The 32 students are the threads. The whole class together is one warp.

The teacher has one hand, so they can write only one instruction on the board at a time, and all 32 students do that same task on their own paper. One order, 32 workers, perfect harmony.

Now the code introduces a split. The teacher writes a condition on the board: If you're in Group A, solve the math problem. If you're in Group B, solve the science problem. Say 16 students fall into A and 16 into B. The road just forked, and the teacher can still only teach one subject at a time.

The cost of divergence

So here is what actually happens, and it's painful. The teacher says: "Group A, do your math now. Group B, sit still and do nothing." So 16 students work while the other 16 sit frozen, idle, wasting their time. Then once Group A finishes, the teacher flips it: "Group A freeze, Group B do your science now." The first half goes idle while the second half works.

Look at the cost. A job that could have finished in one pass now takes two, because the work that used to happen all at once had to be split and run one group after the other. Half the workers are always sitting idle, doing nothing, no matter which path is running. The warp didn't split the work between cores to go faster. It serialized it, and the whole thing runs at roughly half speed.

↳ And if there's an if/else nested inside another if/else, it forks again, and the speed falls off a cliff.

Why CPUs don't care about branching

So the obvious question: the CPU runs if/else all day long and never seems to suffer. Why does branching wreck a GPU but not a CPU? The answer is that a CPU core is a solo runner, not a member of a synchronized dance team, and it carries three pieces of heavy machinery that a GPU core doesn't:

1. Its own brain: Every CPU core has its own hardware to read and understand instructions. It doesn't care what the core next to it is doing. So if one core goes down the if path and another goes down the else path, both just run. Neither has to wait for the other. There's no freezing because there's no group to stay in step with.
2. A guessing engine: A CPU has a powerful branch predictor built in. Before the code even reaches the fork, it guesses which way the branch will go, and that guess is right well over 90 percent of the time.
3. Speculative execution: Having guessed, the CPU doesn't sit and wait. It runs ahead down the predicted path and gets the work done before the branch is even confirmed. If the guess was right, there's zero delay. If it was wrong, it throws the wasted work away and switches, which costs a little but rarely happens.

Why GPUs don't just use branch predictors

So why doesn't a GPU just have all that too? Because of the one constraint that shapes everything about a GPU: space on the chip is finite.

Those guessing engines and independent brains are massive, and a CPU can afford them because it only has a handful of cores. A GPU has thousands. If you bolted a full branch predictor and independent control onto every one of those thousands of cores, the machinery would eat all the room and leave no space for the actual math units.

So the GPU makes the opposite trade on purpose. It keeps the management tiny, packs in as many workers as possible, and accepts that when an if/else shows up, it just stalls and runs the paths one after another.