The Roofline Model: Why LLM Decode Is Memory-Bound

Parts 1 through 8 of this series explained what the model computes. This part is about what it costs. The roofline model is a single chart, built from exactly two hardware numbers, that tells you whether your GPU is starved for data or starved for math. For LLM token generation the answer is always the same, and it is not close.

By the end of this article you will be able to place decode and prefill on that chart yourself, calculate the tokens-per-second ceiling your hardware imposes before you write a line of serving code, and explain to a colleague why a chip with twice the FLOP rate can leave your latency completely unchanged.

The model was written in 2009, for multicore CPUs. It happens to describe transformer inference better than almost anything published since, and it turns a pile of folklore (“batching is free”, “buy a faster GPU”) into one picture with a diagnostic attached.

The frame: two axes and a flat ceiling

The bottom axis: arithmetic intensity

Across the bottom sits arithmetic intensity, measured in FLOPs per byte. It answers one question about an operation: how much math does it perform for each byte it drags out of memory? Thin operations live on the left. A vector add reads two numbers, does one addition, writes one number. Dense operations live on the right. A large matrix multiply reads a tile once and reuses it hundreds of times.

The side axis: achieved performance

Up the side sits achieved performance, in FLOP/s. Not the number on the spec sheet. The rate the chip actually sustains while running your operation. Both axes are log scale, so each gridline is ten times the last.

The question the chart answers is narrow and useful: for an operation of a given density, what is the fastest this hardware could possibly go? Not how fast your code goes. How fast anything could go.

The compute roof

A chip retires a fixed maximum number of multiply-adds per second. On an H100 SXM that is roughly 989 TFLOP/s in bf16 with dense tensor cores. Nothing exceeds it, ever, under any workload, so it draws as a horizontal line: it does not slope, because it does not care about arithmetic intensity. A thin operation and a dense operation face the same ceiling, everything above the line is unreachable, and on its own that fact is useless. Its job is to be one half of a comparison.

The memory roof: why proportionality draws as a diagonal

The chip can only pull bytes out of HBM at a fixed rate. Call it B bytes per second. On an H100 SXM, B is about 3.35 TB/s. Now take an operation with arithmetic intensity I, meaning every byte that arrives carries I FLOPs of work with it. If memory is the thing holding you back, then the performance you actually get is:

achieved FLOP/s = B x I

Bytes per second, times FLOPs per byte. The units cancel to FLOPs per second. Squeeze ten times as much math out of each byte and you get ten times the performance from the exact same delivery rate. The memory system did not get faster. You just asked it to do less work per unit of output.

Why the diagonal sits at 45 degrees

That relationship is a plain proportionality. On linear axes it would be a straight line through the origin. On log axes it draws at 45 degrees, and the reason is worth stating exactly, because this is where readers get lost. Log axes plot the exponent rather than the value, so a step to the right is a multiplication. Move one gridline right and I has gone up by a factor of ten. Performance is B x I, so performance has also gone up by a factor of ten, which is exactly one gridline up. One right, one up, at every point on the line. That is a slope of 1, and a slope of 1 draws at 45 degrees whenever both decades occupy the same physical distance on the page.

So any 45 degree line on this chart is a bandwidth. Its height tells you which one. Faster memory pushes the same diagonal upward without changing its angle. The line is not fitted to measurements. It is B, redrawn in a different coordinate system. This is the same mechanical-sympathy reasoning that governs cache behaviour and memory access patterns on a CPU, scaled up to a device with a 3 TB/s bus and 16,000 arithmetic units to feed.

Take the lower of the two, and the ridge appears

Both limits apply simultaneously. You cannot exceed the arithmetic rate and you cannot exceed the delivery rate. So the real ceiling at any intensity is whichever of the two is lower. On the left the diagonal is lower, so memory decides. On the right the flat line is lower, so compute decides. Keep only the lower value at every point and you get a rising slope that flattens into a ceiling. That is the profile of a pitched roof, and that is where the roofline model gets its name.

The corner where the two lines cross is the ridge point, the intensity at which the two limits are exactly balanced. It falls out of one division:

ridge point = peak FLOP/s / peak bytes/s

For an H100 SXM: 989 TFLOP/s divided by 3.35 TB/s is about 295 FLOPs per byte. Nothing about your model, your framework or your batch size appears in that calculation.

The ridge splits the plane in two

Drop a vertical line from the ridge and the chart divides into two territories. Left of the ridge, bytes arrive too slowly to keep the arithmetic units busy, so the tensor cores finish their work and wait. Right of the ridge, bytes arrive faster than the math can consume them, so the bus has slack.

Read the word “bound” literally. It means bounded by. It names the wall you are pressed against, and therefore the only wall worth attacking. Widening a corridor you are not standing in does nothing. That is the entire diagnostic value of the roofline model, and it is why the chart is worth twenty minutes of your time.

Where LLM work actually lands

The running example for this series is a Llama-3-70B class model: 80 layers, d_model of 8,192, roughly 70 billion parameters, held in bf16. That is about 140 GB of weights. If you want the model-side background on why the forward pass has the shape it does, Part 8 walks the original 2017 architecture box by box.

Two facts drive everything below. First, a forward pass does roughly 2 FLOPs per parameter per token (one multiply and one add). Second, a bf16 parameter occupies 2 bytes. Divide the first by the second and you get a rule you can carry around in your head:

The arithmetic intensity of a bf16 transformer step is approximately the number of tokens processed in that step.

One token in flight gives an intensity near 1. Thirty-two tokens gives roughly 32. Two thousand gives roughly 2,000. If you count fused multiply-add the way vendor spec sheets do, or you add the attention math on top, you get about twice those figures. The factor of two is not what matters here. The gap to the ridge is a factor of several hundred.

Decode at batch 1

To produce one token, the model reads every weight in the network and does about 140 GFLOPs of math with them. Intensity lands at 1 to 2 FLOPs per byte, against a ridge of 295.

Multiply it out. Achieved performance is bandwidth times intensity: 3.35 TB/s times 1 gives 3.35 TFLOP/s, against a peak of 989 TFLOP/s. That is roughly one third of one percent of the chip’s arithmetic capacity. Be generous about the counting and call it one percent. Either way, more than 99 percent of the tensor cores you paid for are idle, and they are idle for the entire duration of every token you generate.

The ceiling bandwidth alone imposes

Because the bytes are the bottleneck, you can compute a hard tokens-per-second ceiling without knowing anything about the software:

max tokens/sec = aggregate memory bandwidth / bytes of weights read per token

140 GB does not fit on one 80 GB H100, so shard it across two with tensor parallelism. Each card now reads its own 70 GB shard per step. At 3.35 TB/s that takes 70 / 3350 = 20.9 ms. Both cards do it at the same time, so a decode step costs about 21 ms and you get roughly 48 tokens per second at batch 1.

That number is an upper bound. No kernel, no compiler, no framework can beat it while the weights stay in bf16 and every one of them has to be read. Real systems land below it, because collective communication between the two cards and reads from the KV cache both add bytes. If your vendor demo shows 40 tokens per second on this setup, they are running at 83 percent of a physical limit and doing well. If a proposal promises 200, ask which of the four assumptions they broke.

Decode at batch 32

Run 32 sequences concurrently and the model reads the same 140 GB, once, and uses it to produce 32 tokens. The bytes did not increase. The math did, by 32 times.

Intensity moves from about 1 to about 32. The point slides right along the diagonal and up the roof. Throughput goes from 48 tokens per second to roughly 1,530. And the wall-clock time per step barely moves, because the step was always waiting on memory and memory is doing the same amount of work. Each individual user still sees a token roughly every 21 ms.

Thirty-two times the throughput for close to zero added latency. That is not a software trick. It is what the geometry of the chart says has to happen.

Prefill

Now process a 2,048 token prompt. Same weights, same 140 GB read, but every weight is used for 2,048 tokens at once. Total math is 2 x 70e9 x 2048, about 287 TFLOPs. Intensity is 287e12 / 140e9, roughly 2,048 FLOPs per byte, which is what the rule of thumb predicted.

That is seven times past the ridge. Prefill sits flat against the compute ceiling with the memory bus half asleep. This is the whole prefill and decode asymmetry, and it is why serving stacks increasingly run the two phases on separate machines with separate hardware. It is also why time-to-first-token and inter-token latency respond to completely different interventions, which matters a great deal if you are streaming responses and optimising perceived latency.

The free lunch, and exactly where it ends

Every step rightward along the diagonal is throughput you get for almost no additional wall-clock time, because the bytes were already moving. Nobody is paying for it. That is a genuinely free lunch and it is the reason continuous batching exists.

The lunch stops at the ridge. Precisely there, and not one intensity unit further. Past the ridge you are compute-bound, the step time now scales with the amount of math, and adding another sequence to the batch buys you queueing delay and nothing else. On an H100 with a bf16 70B model, the ridge sits at a batch of roughly 150 to 300 concurrent sequences depending on how you count FLOPs, so from batch 1 you have a few hundred sequences of headroom to spend.

Two caveats keep this honest. Batching amortises weight reads and does not amortise KV cache reads, because every sequence carries its own KV cache and no two sequences share one. So as batch size and context length grow, KV traffic becomes the dominant memory cost, and the attention operation stays memory-bound no matter how large the batch gets.

The second caveat is why nobody actually reaches batch 300: you run out of memory to hold those KV caches long before you get there, and naive allocation wastes most of what is left. Part 10 covers continuous batching and PagedAttentioncoming 18 Aug, which is the story of what stops you from batching and what vLLM does about it.

Three moves, and no others

Once a workload is placed on the roofline model, the set of things that can possibly help becomes finite and visible. There are three directions, and nothing else moves your point upward.

1. Lift the diagonal. More memory bandwidth. Helps everything left of the ridge, proportionally.
2. Slide right. Fewer bytes for the same math, or more math per byte. Helps until you hit the ridge.
3. Raise the ceiling. More arithmetic capacity. Helps nothing left of the ridge.

Where the real techniques land

Almost every inference optimisation you have heard of is move 2, and almost all of them work by moving fewer bytes. The mechanisms are not interchangeable, so it is worth being exact about each one.

• Batching. More math per byte. FLOPs rise with batch size while bytes stay fixed, so the point slides right. This is the only one of the four that leaves bytes-per-step alone.
• Quantization. Fewer bytes per weight. Weight-only int4 quantization cuts the bytes by four while leaving the FLOP count untouched (the values are expanded back out before the multiply). Intensity goes up four times and the tokens-per-second ceiling goes up four times. This is the single largest batch-1 decode win available, which is exactly what the chart predicts.
• Mixture of experts. Fewer bytes per token, because only the routed experts get read. It is worth being exact: at batch 1 an MoE layer cuts the math by the same factor it cuts the bytes, so the intensity barely moves. What improves is the numerator of the ceiling formula. Fewer bytes per token means more tokens per second from the same bus. At large batch the tokens scatter across experts and you end up reading most of the weights anyway, which is why MoE helps decode more than it helps prefill.
• Kernel fusion and FlashAttention. Fewer bytes, same math. An unfused attention writes the full score matrix out to HBM and reads it back. FlashAttention keeps it in SRAM and never materialises it. The FLOPs are unchanged (slightly higher, in fact, because of recomputation), the HBM traffic collapses, and intensity rises. It is a pure move 2 with no accuracy cost, which is rare.

Speculative decoding is move 2 by a different route

What it converts is k serial memory-bound steps into one wider step, and that is the crucial difference from batching. Batching raises aggregate throughput for many users while holding per-user latency flat. Speculative decoding raises throughput for a single stream, which is the one thing batching cannot do. It also burns extra FLOPs on rejected tokens, which is fine, because on the left of the ridge you had FLOPs to spare. The economics only work while the acceptance rate stays high; a draft model that gets rejected constantly costs you its own bandwidth for nothing.

The expensive mistake this prevents

The mistake is buying move 3 for a workload that lives in move 2’s territory. A chip with twice the FLOP rate and identical bandwidth changes nothing about batch-1 decode. Zero. The ceiling was never the surface you were touching. You will pay a hardware premium, run the benchmark, see the same tokens per second, and spend a fortnight blaming the framework. Ten minutes with the roofline model would have told you the answer before the quote was signed.

Here are real accelerators with the ridge point worked out. Treat these as vendor peak specifications for dense bf16 with no sparsity. Measured numbers typically land 10 to 20 percent below peak, which shifts the ridge slightly but never enough to matter to the argument.

Look at the first two rows together, because that comparison is worth money. The L40S has a higher peak bf16 FLOP rate than an A100 80GB. It also has 42 percent of the bandwidth. For batch-1 decode of a model that fits on both, the A100 is about 2.4 times faster, and the card with the better spec-sheet headline loses badly.

This is also the arithmetic that should sit at the front of any decision about moving LLM workloads on-prem. Before you compare the capital cost of a GPU node against per-token API pricing, compute the bandwidth-implied token ceiling for the model you intend to run at the concurrency you actually expect. Two numbers off a datasheet and one division will tell you whether the box can serve your traffic at all. It is a much cheaper thing to get wrong on a whiteboard than in a purchase order, and Part 11 turns that estimate into a measurement of your own hardwarecoming 19 Aug in an afternoon.

Before you buy anything at all, though, check the cheapest speedup on the list. If you are memory-bound at low batch, raising concurrency costs nothing but scheduler work, and the free lunch is real all the way to the ridge.

Key takeaways

• The roofline model needs two hardware numbers: peak FLOP/s and peak bytes/s. Their ratio is the ridge point, which is where the memory diagonal meets the compute ceiling.
• The memory roof rises because performance equals bandwidth times intensity, and that proportionality draws as a 45 degree line on log axes. A 45 degree line on this chart is a bandwidth.
• The arithmetic intensity of a bf16 transformer step is roughly the number of tokens in that step. Batch 1 decode sits near 1, against ridge points of 150 to 420 on current accelerators.
• Decode at batch 1 uses well under one percent of a modern GPU’s arithmetic capacity. Its tokens-per-second ceiling is memory bandwidth divided by bytes of weights, and no software can exceed it.
• Batching is close to free until the ridge, because the bytes were already moving. Past the ridge it buys only queueing delay.
• Batching, quantization, MoE, kernel fusion and FlashAttention all slide you rightward, and all of them work by moving fewer bytes. Speculative decoding does the same thing for a single stream by verifying several tokens per weight read.
• Doubling the FLOP rate does nothing for batch-1 decode. Buy bandwidth for chat, buy arithmetic for bulk prefill.

Frequently asked questions

Is LLM inference memory-bound or compute-bound?

Both, depending on the phase. Decode (generating tokens one at a time) is heavily memory-bound, with an arithmetic intensity near 1 FLOP per byte against GPU ridge points of 150 to 420. Prefill (processing the prompt) is compute-bound, because thousands of tokens share a single read of the weights.

What is the ridge point of a GPU?

The ridge point is peak FLOP/s divided by peak memory bandwidth, expressed in FLOPs per byte. It is the arithmetic intensity at which the two limits balance exactly, and it is a pure property of the hardware with no dependence on your model or framework. An H100 SXM sits near 295 FLOPs per byte.

How do I calculate the maximum tokens per second for a model on my GPU?

Divide your aggregate memory bandwidth by the bytes of weights read per token. A 70B model in bf16 is about 140 GB, so two H100 SXM cards at 3.35 TB/s each give roughly 48 tokens per second at batch 1. That is a hard upper bound that no serving software can beat.

Does a faster GPU make single-user LLM generation faster?

Only if the newer chip has more memory bandwidth. Batch-1 decode never touches the compute ceiling, so a card with twice the FLOP rate and the same bandwidth will produce identical tokens per second. Compare GB/s, not TFLOP/s, when latency for one user is what you care about.

Why does batching increase throughput without increasing latency?

Because the weight bytes were already being moved. At batch 1 the arithmetic units sit idle waiting for memory, so adding more sequences fills that idle time with useful math instead of extending the step. The effect holds until arithmetic intensity reaches the ridge point, after which extra batch adds queueing delay.

Does quantization help because of fewer FLOPs or fewer bytes?

Fewer bytes, in the decode case. Weight-only int4 quantization leaves the FLOP count essentially unchanged and cuts the bytes read per token by four, which raises both the arithmetic intensity and the bandwidth-implied token ceiling by the same factor. That is why it is the largest single win available for memory-bound decode.

Sources and further reading

• Roofline: An Insightful Visual Performance Model for Multicore Architectures, Williams, Waterman and Patterson, the original 2009 paper.
• Efficiently Scaling Transformer Inference, Pope et al., which works the arithmetic intensity of transformer inference through in detail.
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, the canonical example of optimising an operation by cutting HBM traffic.
• Fast Inference from Transformers via Speculative Decoding, Leviathan, Kalman and Matias.
• Efficient Memory Management for Large Language Model Serving with PagedAttention, the vLLM paper.
• vLLM documentation, for the batching and scheduling behaviour discussed in Part 10.