People say put it on the GPU like it's a magic speed button. Sometimes you press it and your code gets slower than a cheap laptop. Understanding why comes down to one fact almost nobody internalizes at first. Your computer is not one machine. It's two computers in one box, each with its own separate memory, joined by a single slow bridge. And most of the time the bridge, not the math, decides how fast you go.
↳ Start with the divide itself. The CPU owns one pile of memory, the system RAM. The GPU owns a completely different pile, its own dedicated VRAM. These are not the same memory with different labels. They are different technologies built for opposite goals, and that difference is the whole story.
System RAM is tuned for low latency. The CPU mostly does one branchy, sequential thing at a time, so it wants a single item back instantly. VRAM is tuned for raw throughput. The GPU has thousands of cores that all want a flood of data at the same instant, so VRAM is built to deliver a huge gush rather than a quick single hit.
And here is the question that exposes why you can't just use the fast one everywhere. If low latency means getting a byte instantly, shouldn't the GPU want that too? Wouldn't its threads compute faster pulling from the snappy CPU memory?
With one thread, yes. But a GPU has ten thousand. Picture the low-latency memory as a vending machine that drops one snack instantly — wonderful for one person. But line up ten thousand threads at that one machine and thread number ten thousand waits through ten thousand presses. The instant response is destroyed by the queue. Now picture the GPU memory as a dump truck that takes longer to arrive but then drops ten thousand snacks on the floor at once, so every thread grabs its own at the same moment. The slower start wins because the throughput serves everyone together.
↳ You physically cannot build one memory that does both at once, so the two computers keep two different memories. Which also kills the dream of one big shared pool — because the fast GPU memory only hits its enormous speed when it's physically stacked against the GPU die, and you can't put it next to the CPU too.
Now the bridge between these two worlds, and it is the villain of this whole story. The two memories are joined by a connection called the PCIe bus. Inside the GPU, data moves at well over a thousand gigabytes per second — an internal superhighway. The bridge between CPU and GPU moves only a few tens of gigabytes per second. So the bridge is something like fifteen to thirty times narrower than the highway it feeds. Every single byte that crosses from the CPU side to the GPU side squeezes through that choke point.
Because the two memories are separate, you can't just compute. You have to do a three-step dance every time. First, copy the input data across the narrow bridge into GPU memory. Second, launch the work, which runs on data now sitting in GPU memory. Third, copy the result back across the bridge. Copy in, compute, copy out. And those three steps are serial — you cannot compute on data that's still crossing the bridge.
Now the trap that catches everyone, and it has a name worth knowing: Amdahl's Law. Say the actual GPU math takes 1 millisecond — blazing. But moving the data across the narrow bridge takes 100 milliseconds. Your real wall-clock time is 101 milliseconds, almost entirely bridge. Meanwhile a plain CPU, with the data already sitting right next to it, might finish the whole job in 20. So the GPU, despite doing the math fifty times faster, made the program five times slower.
↳ The lesson is brutal and simple. What matters is the total time on the clock, not how fast the math units can theoretically go. A GPU starved for data, waiting on the bridge, is not helping you.
Only when the compute time dwarfs the transfer time. If you cross the bridge once and then grind on that data for thousands of steps without going back, the one-time crossing cost becomes a rounding error against all the math. The ratio of math to bytes moved has to be high. And that points straight at the defenses.
The first defense is reuse — do enormous math per byte. The perfect GPU job uploads data once, performs a mountain of operations on every element, and sends back a tiny result. Training a neural network is exactly this: you load the weights and a batch, then run an enormous number of operations on them without touching the CPU again. Multiplying two large matrices does work that grows with the cube of the size while the data only grows with the square, so the bigger the problem, the more the math drowns out the transfer. High math-to-bytes is what makes the bridge crossing worth it.
The second defense is overlap, called streams. The idea is: why let the bridge sit idle while the GPU computes, and why let the GPU sit idle while the bridge copies? So you chop the data into chunks and pipeline them. While the GPU computes on chunk two, the bridge is already hauling chunk three in, and hauling chunk one's result back out. Done well, the transfer hides behind the compute, and your total time becomes the longer of the two instead of their sum.
But this only works if the compute per chunk is big enough to cover the next chunk's transfer. If your math is too light, the GPU finishes a chunk and then sits waiting for the next one to arrive anyway. You didn't hide the transfer — you just chopped your idle time into smaller pieces.
The third thing people reach for is unified memory, where you use one pointer that both the CPU and GPU can touch, and the system moves data behind the scenes so you don't write the copies yourself. And the question it begs is: if the system handles it automatically, isn't the problem solved?
It isn't. Unified memory is a convenience in the code, not a change in the physics. Under the hood the pages still cross the same narrow bridge. When the GPU touches data still living on the CPU side, the hardware stops dead, faults, and the system scrambles to copy that page across before resuming. If your access pattern jumps around randomly, you trigger a storm of page faults and end up slower than if you had just done one clean bulk copy yourself. The syntax got easier. The bridge did not get wider.
And to see the bridge at its absolute worst, consider the tempting shortcut: why copy at all? Why not let the GPU just reach across the bridge for each value as it computes? Because then every single math instruction — something that should take a cycle or two — now waits on a full round trip across the slowest connection in the machine. Instead of hiding memory delay, you've wired the slowest bus directly into the math loop, and the cores spend nearly all their time frozen, waiting for signals to crawl across the motherboard. The only winning move is to pay the crossing cost once, get the data into fast GPU memory, and then feast on that internal superhighway.
Your machine is two computers with two separate memories built for opposite goals, joined by a bridge far narrower than either side's internal speed. Every byte that crosses is expensive, the work is a serial copy-compute-copy dance, and a fast math kernel trapped behind a slow transfer is slower overall than never using the GPU at all.
The GPU only wins when the math per byte is huge, or when you cleverly hide the transfer behind compute.
The bottleneck is almost never the math.
It's moving the data to where the math can happen.
The bridge is the real enemy.