Why C++ won't auto-compile into GPU code

*the anatomy of a gpu thread*

If you write GPU code thinking a thread is a tiny worker the way it is on a CPU, you will be confused by everything the hardware does. Because on a GPU, a thread is not a worker at all. It's an illusion.

Understanding what a thread actually is at the silicon level, and why NVIDIA wraps them in blocks and warps, is the thing that separates people who write fast GPU code from people who write code that mysteriously crawls. Start with the hard truth: there is no physical thing called a thread inside a GPU.

The Illusion: What a Thread Actually Is

On a CPU, a thread is a heavy object. It has its own stack, its own registers, its own scheduling brain, and the operating system treats it as a real entity.

On a GPU, a thread is just two things. An index number, like a seat number, and a program counter pointing at which line of code is running. That's it.

When you launch a kernel with ten thousand threads, you are not creating ten thousand workers. You are issuing one instruction and telling the hardware to run it across ten thousand different pieces of data. The thread is a data lane, not a person.

The Structure: Index Math

So how does a thread find its own data, if it's just an index? With pure arithmetic, no lookup, no scheduling. Every thread runs the identical code, so each one computes its own address from its coordinates.

Its global position is its block number times the size of a block, plus its position inside that block. Block one, thread two, in blocks of four, lands at global position six, so it grabs the sixth item.

The silicon works this out in a single step. No operating system involved, just math.

The Barrier: Why Blocks Exist

Now the question that actually matters. If the real work is done by math units inside the chip, why did NVIDIA bother inventing blocks? Why not just launch a flat pile of threads? The answer is that three hard silicon limits would break if they did:

• First, the wiring problem. Imagine ten thousand workers in one hall who can all talk to each other. To wire every worker to every other worker, the number of connections explodes roughly with the square of the worker count. A million workers would need a trillion wires. The chip would be nothing but copper. So NVIDIA partitions the threads into small cabins, the blocks. Inside a block, the threads sit together and share a fast scratchpad of memory to talk cheaply. But a thread in one block cannot reach into another block's scratchpad. The wiring only has to be built locally.
• Second, the resource problem. Each processing unit on the chip has a fixed pool of registers. A block acts like a booking ticket. The hardware looks at a block and knows exactly how many registers and how much scratchpad it needs, so it can ask: do I have room for this whole block right now? If yes, it admits it. If not, it waits in line.
• Third, the scaling problem. You write your code once as a grid of many blocks. A small laptop GPU with two processing units runs those blocks a few at a time. A giant data-center GPU with a hundred units runs a hundred blocks at once. Same code, no changes. The block is the unit that lets one program automatically scale.

The Warp: Lockstep and Branching

Now go one level deeper. The real unit the hardware runs is not the thread and not the block. It's the warp, a fixed group of 32 threads. When your block of 256 threads lands on a processing unit, the hardware silently chops it into 8 warps of 32.

And a warp moves in lockstep. The scheduler fetches one instruction and broadcasts it to all 32 lanes at once, each working its own data. The threads were never independent. They're bolted together in groups of 32.

Why 32 specifically? It's a balance. Make the group too small, and you need more schedulers eating silicon. Make it too big (say 64), and the bigger the group forced to march in step, the more likely some of them want to go different ways.

The Silent Killers: Registers and Occupancy

Now the part that trips up almost everyone writing real kernels: registers. Each thread gets a slice of that fast register pool (1 cycle to access). But the pool is fixed.

If a thread needs more registers than the hardware can give it, the leftover variables get dumped out to a place misleadingly called local memory. Don't let the name fool you. Local memory physically lives out in the slow main memory. So a variable that took 1 cycle to reach now takes 200 or more. This is register spilling, turning your bullet train into a bicycle.

And registers feed directly into the deepest trap of all: occupancy. Occupancy is how full you keep the processing unit with active warps. Here's the math: say a unit can hold 65,536 registers and run at most 64 warps. If each thread uses 32 registers, all 64 warps fit. Full occupancy.

Now you add a few variables, and each thread needs 64 registers. Suddenly only half as many threads fit in the pool. Occupancy drops to 50%, and math units sit idle. You didn't change how many threads you launched; you just used more registers, and the hardware choked.

The Answer: Why the Compiler Can't Save You

So let's go back to the title of this chapter: why can't you just take standard C++ code and let a clever compiler turn it into GPU instructions?

Because standard C++ was designed for the "Heavy Engine." The language assumes threads are independent, memory is flat, and branching is practically free. A compiler translating a standard for loop has no way to know how to chop your data into a grid, how to size the blocks to avoid killing register occupancy, or which variables need to sit in the shared scratchpad to avoid melting the memory bus.

If a compiler simply unrolled a C++ loop and dumped it onto the GPU's "Conveyor Belt", it would shatter the lockstep execution, spill registers to slow memory, and stall out. The compiler lacks the vocabulary to map standard code onto this radically different hardware.

To write fast GPU code, you need a way to explicitly tell the hardware about blocks, grids, and shared memory. You need a new vocabulary. That vocabulary is CUDA.