Compute-bound vs. memory-bound

*worked out from first principles*

There is a question that decides whether buying a faster chip is worth the money or a complete waste of it: will this expensive new hardware actually make my code faster?

The answer is sometimes yes, and sometimes nothing changes at all. The roofline model is the visual picture that tells you which one it will be.

↳ To read it, you only need to look at two physical numbers about any chip: FLOPS and bandwidth.

The raw metrics: FLOPS and Bandwidth

First, let's establish the two players in this game:

• FLOPS (Floating Point Operations Per Second): The math speed. It means how much math all the compute units combined can finish in one second. Think of a factory with workers carving designs into bricks. FLOPS is their total carving speed. A chip might top out at, say, 100 operations a second. That is its hard physical ceiling. No matter how much data you pour in, the workers cannot push past 100.
• Bandwidth: The supply speed. The microscopic wires carrying data into the chip. It means how much data can arrive in one second. This is the width of the road feeding the factory.

A workload is held back by one of these two, never a third thing. The roofline graph draws this limit as a house, with a sloped wall on the left rising up into a flat roof on top. Where your code lands on that house is the whole story.

The Slope: Memory-Bound Territory

Start with the slope. This is where memory-bound code lives.

Memory-bound means the data is huge but the math on it is light. Picture a job that grabs a number, adds 1, and sends it back, repeating this across a mountain of data. The workers can carve 100 bricks a second, but the thin road only delivers 20 bricks a second. So the workers finish those 20 instantly and then sit idle, because nothing more has arrived. They are nowhere near their 100 ceiling. The road is choking them, not their own hands.

↳ The punchline of the slope: you buy a new chip with double the FLOPS, so the workers can now carve 200 a second. What changed? Nothing. The road is still delivering 20 bricks a second. The workers just sit idle twice as fast. Zero speedup.

On the slope, more math power is money set on fire, because math was never the problem.

The Flat Roof: Compute-Bound Territory

Now look at the flat roof. This is where compute-bound code lives.

Compute-bound means the data is small but the math is brutal. Picture one tiny brick that needs 500 separate carvings. Here, the road is wide enough to deliver 1000 bricks a second, far more than needed, so there is no traffic jam at all. Data is sitting there ready and waiting.

The workers are running flat out at their full 100 operations a second and cannot go faster. They have hit the ceiling, the flat roof of the graph. It is their own hand speed stopping them now, not the road.

↳ And the punchline of the roof is the opposite: you buy a chip with double the FLOPS, workers go from 100 to 200, and because the data was already sitting there ready, the whole program runs genuinely 2x faster. Here, the money was well spent, because math was the actual bottleneck.

So the same hardware upgrade is either completely useless or transformative, and which one depends entirely on where your code sat on the house before you bought anything.

The Trap: Shifting Bottlenecks

Now the deepest twist, the one that traps people. You assume your code will always stay on the roof, so faster math always pays off.

But here is what actually happens: when you buy a chip with more FLOPS, the roof itself rises higher. The workers got faster, but the wires carrying data are bound by physics that did not improve. A program that was compute-bound on the old chip can suddenly become memory-bound on the new one.

↳ The workers raced ahead and left the road behind. You paid for more carving speed and walked straight into a delivery jam you didn't have before.

That is why raising a chip's math power without thinking about its bandwidth can be the biggest waste of money there is. A chip running at 20 percent utilization isn't a chip that is lazy. It is a chip whose workers are standing around waiting for a jammed road to clear.

Core vs. FLOPS

One thing is worth separating cleanly here, because they easily blur together: a core and FLOPS are not the same.

A core is the physical worker, an actual unit etched into the silicon that you could point to. FLOPS is the speed score, how much math all those workers manage per second. You can raise FLOPS two ways: by adding more workers or by making each worker faster. The worker is the physical object, and FLOPS is its performance.

When code hits the flat roof, it means every core is running its hands at full speed and there is simply no faster gear left.