Benchmark Your Own LLM Serving Stack: Two Measurements, One Afternoon

Ten parts of this series have argued from first principles. This one hands you two scripts and asks you to stop taking anyone’s word for it. An LLM serving benchmark that you ran yourself, on your own card, with your own model and your own flags, is worth more than every published throughput chart you have ever read, because those charts were produced on hardware you do not own with settings nobody wrote down.

The whole design is one idea. You take two measurements. The first asks what the silicon can do. The second asks what your serving stack actually achieves. The second will land far short of the first, and the reason it falls short is the finding.

By the end of this you will have a measured bandwidth number, a measured peak, a measured ridge point, three curves showing where your stack stops scaling, and a one sentence diagnosis of which wall you hit. It is an afternoon, and most of that is waiting.

Why two measurements and not one

Script 01 runs a bare matrix multiply. No model, no KV cache, no server, no scheduler, no tokenizer. Just a GEMM on the GPU, swept across a range of shapes. It finds the ridge point that the roofline model from Part 9 predicts: the arithmetic intensity at which the card stops being limited by memory bandwidth and starts being limited by arithmetic.

Script 02 runs a real model behind a real server with a real KV cache and real concurrent requests, ramping concurrency from 1 to 128 and recording what you actually get at each level.

The two numbers will not agree. They are not supposed to. The gap between them is the entire product of the exercise, and a useful LLM serving benchmark is really just a careful description of that gap.

Here is why this matters commercially, stated plainly. Anyone can quote a datasheet. Almost nobody can say “here is where my stack actually saturates, and here is why it is not where the hardware says it should be.” That sentence, backed by two plots, is a deliverable. It is also the difference between advising and guessing when somebody tells you their inference endpoint feels slow.

Script 01: what the silicon can do

The trick, explained properly

The probe runs exactly one shape: (B, 8192) @ (8192, 8192). That is a transformer projection. It is the shape of W_Q, of W_O, of any square projection in a model with d_model of 8,192. It sweeps B from 1 to 2048 across 17 batch sizes.

The reason one shape is enough is worth doing on paper. In bf16, every element is 2 bytes. For a GEMM of (B, K) @ (K, N):

FLOPs = 2 * B * K * N
bytes = 2 * (K*N  +  B*K  +  B*N)
             weights  input   output

intensity = FLOPs / bytes
          = B*K*N / (K*N + B*K + B*N)

divide top and bottom by K*N:

intensity = B / (1 + B/N + B/K)

with K = N = 8192:

intensity = B / (1 + B/4096)

The weight matrix is 8192 by 8192, which is 67 million elements. The activations are B by 8192, which is only 8,192 elements per row. Until B gets into the hundreds, the weights dominate the bytes read and the denominator barely moves. So arithmetic intensity is approximately B.

That is the trick. Sweeping batch size is sweeping arithmetic intensity. The decode-batching argument from Part 9, the one that says extra concurrent users are nearly free while you are memory-bound, stops being a claim and becomes a measurement you can plot.

The probe

This is complete and self-contained. It needs PyTorch and a CUDA device and nothing else. Timing uses CUDA events rather than a host-side clock, because a host-side clock measures kernel launches unless you synchronise carefully, and the warmup pass exists so you are not measuring the first-call compilation and allocator behaviour.

#!/usr/bin/env python3
"""Measure the empirical roofline of one GPU.

One GEMM shape, (B, 8192) @ (8192, 8192), which is a transformer
projection. Sweeping B sweeps arithmetic intensity, because intensity is
approximately B while K and N are large. Writes roofline_GPUNAME.json.
"""
import json
import torch

K = N = 8192
DTYPE = torch.bfloat16
BATCHES = [1, 2, 4, 8, 16, 24, 32, 48, 64, 96, 128, 192, 256, 384, 512, 1024, 2048]


def timed(fn, warmup=5, iters=20):
    """Seconds per call. CUDA events, so we time the device, not the launch."""
    for _ in range(warmup):
        fn()
    torch.cuda.synchronize()
    start = torch.cuda.Event(enable_timing=True)
    end = torch.cuda.Event(enable_timing=True)
    start.record()
    for _ in range(iters):
        fn()
    end.record()
    torch.cuda.synchronize()
    return start.elapsed_time(end) / 1e3 / iters


def main():
    assert torch.cuda.is_available(), "no CUDA device visible"
    props = torch.cuda.get_device_properties(0)
    el = torch.tensor([], dtype=DTYPE).element_size()

    # 1. bandwidth: one large device to device copy, bytes read plus written
    n = (512 * 1024 * 1024) // el
    src = torch.randn(n, device="cuda", dtype=DTYPE)
    dst = torch.empty_like(src)
    t = timed(lambda: dst.copy_(src))
    bw = 2 * n * el / t / 1e12                 # TB/s
    del src, dst
    torch.cuda.empty_cache()

    # 2. intensity sweep: one fixed weight matrix, growing batch
    w = torch.randn(K, N, device="cuda", dtype=DTYPE)
    rows = []
    for b in BATCHES:
        try:
            a = torch.randn(b, K, device="cuda", dtype=DTYPE)
            t = timed(lambda: a @ w, warmup=3, iters=10)
        except torch.cuda.OutOfMemoryError:
            torch.cuda.empty_cache()
            break
        flops = 2 * b * K * N
        moved = (K * N + b * K + b * N) * el
        rows.append({"batch": b, "intensity": flops / moved,
                     "tflops": flops / t / 1e12, "latency_ms": t * 1e3})
        print("B=%-5d I=%8.1f  %7.1f TFLOP/s  %8.3f ms"
              % (b, rows[-1]["intensity"], rows[-1]["tflops"], t * 1e3))
        del a

    peak = max(r["tflops"] for r in rows)
    ridge = peak / bw
    knee = min((r for r in rows if r["tflops"] >= 0.9 * peak),
               key=lambda r: r["batch"], default=None)

    print("\n%s   %.0f GB   %d SMs"
          % (props.name, props.total_memory / 1e9, props.multi_processor_count))
    print("bandwidth        %.0f GB/s" % (bw * 1000))
    print("peak observed    %.1f TFLOP/s" % peak)
    print("empirical ridge  %.0f FLOPs/byte" % ridge)
    print("90%% of peak at   batch %s" % (knee["batch"] if knee else "not reached"))

    out = "roofline_%s.json" % props.name.replace(" ", "_")
    json.dump({"gpu": props.name, "dtype": str(DTYPE), "bandwidth_tbs": bw,
               "peak_tflops": peak, "ridge_flops_per_byte": ridge,
               "knee_batch": knee["batch"] if knee else None, "sweep": rows},
              open(out, "w"), indent=2)
    print("wrote " + out)


if __name__ == "__main__":
    main()

pip install torch matplotlib

CUDA_VISIBLE_DEVICES=0 python 01_roofline_probe.py
CUDA_VISIBLE_DEVICES=1 python 01_roofline_probe.py

Two minutes per card. Run each card separately before you run both together. If two identical cards differ by more than a few percent, you have a power limit or a thermal problem, and you want to know that before you quote numbers to anyone. Check with nvidia-smi -q -d POWER,TEMPERATURE.

Reading the three numbers

Everything the probe prints is supporting detail for three values.

Measured bandwidth. Expect 10 to 20 percent below the datasheet. A card rated at 1,792 GB/s will typically measure somewhere in the 1,450 to 1,650 range on a plain copy. That shortfall is real. It is refresh overhead, ECC where present, and the fact that a copy kernel is not a synthetic best case. Quote yours, not the spec.

Measured peak. Same story, and one extra trap: vendor tensor-core FLOP/s figures are frequently quoted with 2:4 structured sparsity, which doubles the headline. If your dense bf16 measurement is about half the marketing number, that is why. Halve the datasheet figure before you compare.

Measured ridge point. This is just peak / bandwidth, and the units work out to FLOPs per byte. If you measure 1,500 GB/s and 170 TFLOP/s, the ridge sits at 170,000 divided by 1,500, which is about 113 FLOPs per byte. Because intensity is approximately B, that means the knee lands near batch 116 once you undo the correction term. Below that batch you are memory-bound. Above it you are compute-bound.

One comparison is worth making explicitly. A datacentre card like an H100 has a much higher ridge than a consumer card, because its arithmetic scaled up faster than its bandwidth did. Lower is better here. A lower ridge means the card is more balanced, and that decode saturates its compute at a smaller batch, which is a favourable property for a serving box. Compute both sides with the same convention, dense against dense.

The shape you should see

Plot achieved TFLOP/s against intensity on log-log axes. The measured points should climb along a 45 degree line, then bend flat. The bend is the ridge.

Script 02: what the serving stack actually achieves

Four metrics, defined precisely

Most attempts to benchmark LLM inference go wrong here, because people conflate these four metrics and then argue about results that were never measuring the same thing. Throughput and latency are different products sold to different buyers. A chat product sells TPOT. A batch summarisation pipeline sells throughput. They trade against each other, and the trade is the whole reason a serving queueing model exists.

Report p50 and p99 for the latency metrics, never the mean. The mean hides the tail, and the tail is where the truth lives. A p50 TTFT of 200 ms with a p99 of 4 seconds is a broken system that averages out to looking fine. This is also why streaming responses change the felt experience so much: TTFT is what the user notices first, and TPOT is what they notice after that.

Starting the server and running the sweep

pip install aiohttp vllm

# terminal 1
CUDA_VISIBLE_DEVICES=0 vllm serve <model> \
    --port 8000 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 4096 \
    --enable-prefix-caching

# terminal 2
python 02_serving_sweep.py --model <model> \
    --prompt-tokens 512 --output-tokens 128

Ten to twenty minutes. Keep terminal 1 visible while it runs. The log tells you things the numbers never will, and you are going to need it in a moment.

The concurrency sweep client

This hits any OpenAI-compatible /v1/completions endpoint. It streams, so it can separate TTFT from TPOT. It forces a fixed output length so the levels are comparable, and it randomises every prompt from a 17 word list so prefix caching cannot quietly inflate the result.

#!/usr/bin/env python3
"""Ramp concurrency against an OpenAI-compatible /v1/completions endpoint
and record TTFT, TPOT and output throughput at each level."""
import argparse, asyncio, json, random, time
import aiohttp

WORDS = ["analysis", "capital", "ledger", "transfer", "compliance", "risk",
         "audit", "control", "policy", "exposure", "settlement", "reserve",
         "liquidity", "covenant", "collateral", "custody", "mandate"]


def make_prompt(n_tokens):
    """Randomised, so prefix caching cannot silently inflate the numbers.
    English runs about 1.3 tokens per word."""
    return " ".join(random.choice(WORDS) for _ in range(int(n_tokens / 1.3)))


async def one_request(session, url, model, prompt, max_tokens, out):
    payload = {
        "model": model,
        "prompt": prompt,
        "max_tokens": max_tokens,
        "min_tokens": max_tokens,   # fixed length, so levels are comparable
        "ignore_eos": True,         # same reason
        "temperature": 0.0,
        "stream": True,
    }
    t0 = time.perf_counter()
    t_first, n = None, 0
    try:
        async with session.post(url, json=payload) as resp:
            if resp.status != 200:
                out.append({"error": "HTTP %d: %s" % (resp.status,
                                                      (await resp.text())[:200])})
                return
            async for raw in resp.content:
                line = raw.decode("utf-8").strip()
                if not line.startswith("data: "):
                    continue
                body = line[6:]
                if body == "[DONE]":
                    break
                if not json.loads(body)["choices"][0].get("text"):
                    continue
                if t_first is None:
                    t_first = time.perf_counter()
                n += 1
    except Exception as e:
        out.append({"error": repr(e)})
        return

    t_end = time.perf_counter()
    if t_first is None or n < 2:
        out.append({"error": "no tokens returned"})
        return
    out.append({
        "ttft_ms": (t_first - t0) * 1e3,
        "tpot_ms": (t_end - t_first) * 1e3 / (n - 1),
        "e2e_s": t_end - t0,
        "tokens": n,
    })


async def run_level(url, model, conc, p_tok, o_tok, rounds):
    results = []
    conn = aiohttp.TCPConnector(limit=conc + 8)
    timeout = aiohttp.ClientTimeout(total=900)
    async with aiohttp.ClientSession(connector=conn, timeout=timeout) as s:
        # warm the server at this level, throw the numbers away
        await asyncio.gather(*[
            one_request(s, url, model, make_prompt(p_tok), 16, [])
            for _ in range(min(conc, 4))])
        t0 = time.perf_counter()
        for _ in range(rounds):
            await asyncio.gather(*[
                one_request(s, url, model, make_prompt(p_tok), o_tok, results)
                for _ in range(conc)])
        wall = time.perf_counter() - t0
    return results, wall


def pct(xs, p):
    xs = sorted(xs)
    return xs[min(int(len(xs) * p / 100), len(xs) - 1)]


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--base-url", default="http://localhost:8000")
    ap.add_argument("--model", required=True)
    ap.add_argument("--prompt-tokens", type=int, default=512)
    ap.add_argument("--output-tokens", type=int, default=128)
    ap.add_argument("--rounds", type=int, default=2)
    ap.add_argument("--levels", default="1,2,4,8,16,24,32,48,64,96,128")
    ap.add_argument("--out", default="serving_sweep.json")
    args = ap.parse_args()

    url = args.base_url.rstrip("/") + "/v1/completions"
    print("%5s %9s %9s %9s %9s %9s %4s"
          % ("conc", "tok/s", "TTFTp50", "TTFTp99", "TPOTp50", "TPOTp99", "err"))

    rows = []
    for c in [int(x) for x in args.levels.split(",")]:
        res, wall = asyncio.run(run_level(url, args.model, c, args.prompt_tokens,
                                          args.output_tokens, args.rounds))
        ok = [r for r in res if "error" not in r]
        if not ok:
            print("%5d  all failed: %s" % (c, res[0]["error"][:60]))
            break
        ttft = [r["ttft_ms"] for r in ok]
        tpot = [r["tpot_ms"] for r in ok]
        row = {
            "concurrency": c,
            "throughput_tok_s": sum(r["tokens"] for r in ok) / wall,
            "ttft_p50_ms": pct(ttft, 50), "ttft_p99_ms": pct(ttft, 99),
            "tpot_p50_ms": pct(tpot, 50), "tpot_p99_ms": pct(tpot, 99),
            "e2e_p50_s": pct([r["e2e_s"] for r in ok], 50),
            "errors": len(res) - len(ok), "completed": len(ok),
        }
        rows.append(row)
        print("%5d %9.1f %9.0f %9.0f %9.1f %9.1f %4d"
              % (c, row["throughput_tok_s"], row["ttft_p50_ms"],
                 row["ttft_p99_ms"], row["tpot_p50_ms"], row["tpot_p99_ms"],
                 row["errors"]))

    if not rows:
        print("no successful levels. Is the server running?")
        return

    best = max(rows, key=lambda r: r["throughput_tok_s"])
    base = rows[0]["tpot_p50_ms"]
    knee = next((r for r in rows if r["tpot_p50_ms"] > 1.5 * base), None)
    print("\npeak throughput   %.0f tok/s at concurrency %d"
          % (best["throughput_tok_s"], best["concurrency"]))
    print("batch-1 TPOT      %.1f ms (%.0f tok/s for a single user)"
          % (base, 1000 / base))
    print("TPOT +50%% at      %s"
          % ("concurrency %d" % knee["concurrency"] if knee
             else "never reached. Raise --levels, you did not saturate."))

    json.dump({"config": vars(args), "rows": rows}, open(args.out, "w"), indent=2)
    print("wrote " + args.out)


if __name__ == "__main__":
    main()

Reading the three curves and finding the wall

Plot throughput, TPOT and TTFT against concurrency as three panels side by side, one point per level from the sweep JSON. Each panel answers a different question and each has a shape you can predict before you look.

Throughput climbs steeply, then flattens. The flattening point is your real operating ceiling. Every request you admit past it buys queueing delay and nothing else. Note that the peak throughput number and the peak throughput concurrency are two different findings, and the second one is the more useful one.

TPOT stays nearly constant through the early levels. This is the important one. Going from 1 concurrent user to 8 should cost you almost nothing per token, because during decode the GPU spends its time reading weights out of memory and those weights get read once for the whole batch. That flat stretch is Part 9 made visible on your own hardware, and it is the result that surprises people who have never measured it.

TTFT rises, and the p99 rises faster than the p50. That divergence is the signal. Prefill compute affects both percentiles roughly equally. Queueing affects the tail first.

The sweep does not compute goodput for you, and it does not need to. Pick your SLO, then walk the rows: any level whose p99 TTFT and p99 TPOT both sit inside the budget has goodput equal to its throughput, and the first level that fails the budget is where usable capacity stopped, whatever the throughput column says.

Which wall did you hit

When TPOT starts climbing you have hit one of two walls, and they need completely different fixes. The client numbers cannot separate them, because both walls bend the same curve the same way. The server log separates them in about thirty seconds.

1. Look at the vLLM log for the level where TPOT broke. Not the client output. The server log.
2. If you see preemption warnings (messages about a sequence group being preempted, or about cache blocks being swapped or recomputed), you ran out of KV cache. This is a memory problem, and it is the exact failure mode continuous batching and PagedAttention in Part 10 are designed to postpone. The scheduler admitted more sequences than it had blocks for, and it is now evicting and recomputing. The fixes are memory fixes: lower --max-model-len, raise --gpu-memory-utilization, enable KV cache quantisation, or serve a smaller or more aggressively grouped model.
3. If the log is clean and GPU utilisation sits near 100 percent, you reached the compute knee. There is nothing to fix. You are using the card fully and the ridge point from script 01 should roughly agree with where this happened. If it does not agree, that disagreement is itself a finding worth chasing.
4. Separately, check p99 TTFT against p50 TTFT. If p99 is pulling away from p50 while TPOT is still flat, that is queueing, not compute. Requests are waiting to be admitted. The fix is an admission control or concurrency cap in front of the server. A bigger GPU will not help.

A client says the endpoint is slow. Anyone can run a load test. Telling them which of the three it is, KV cache or compute or queueing, from the log, in under a minute, is what the engagement is actually paying for.

Methodology hygiene, and what voids a run

Treat these as rules. Break any of them and your LLM serving benchmark produces a number that looks authoritative and means nothing.

• Randomise the prompts. If every request shares a prefix, prefix caching will serve most of the prefill from cache and your TTFT will look wonderful. Test that case deliberately if your workload has shared prefixes. Do not let it happen by accident.
• Force fixed output lengths. Without min_tokens and ignore_eos, some requests stop early and the levels are no longer comparable, because a level that happened to generate fewer tokens looks faster.
• Warm up before measuring. The first requests at any concurrency level pay for CUDA graph capture, allocator growth and scheduler ramp. Discard them.
• Measure one card before you measure two. Two cards that disagree by more than a few percent on the same probe means a power or thermal issue, and every multi-GPU number you take after that inherits the problem.
• Report the context alongside every number. Model, quantisation, dtype, context length, prompt and output token counts, vLLM version, driver version, and every server flag. A throughput figure without that list is not a measurement. It is a rumour.

Honest caveats

These scripts are a starting point. They are not a finished harness, and you should expect to fix something small on the first run.

The most likely thing to break: if your vLLM version rejects min_tokens or ignore_eos, it predates them. Drop both fields from the payload. Output lengths will then vary a little more between requests, which is survivable as long as you say so when you report the numbers.

Also: results are specific to your model, your card, your driver version, your vLLM version and your flags. Change any one of those and rerun. A benchmark is a snapshot of a configuration, and treating it as a property of the hardware is how people end up quoting numbers that stopped being true two releases ago.

Variations worth running

Each one is a single flag change, and each answers a question you would otherwise have to guess at.

Run the last one early. Without NVLink, tensor parallelism pays an interconnect tax on every single layer, because each layer needs an all-reduce across the cards and that traffic crosses PCIe. Two independent replicas behind a load balancer share nothing and pay nothing, so they will usually win on throughput. They lose on maximum model size and on single-request latency, which is the real trade. “Probably” becomes “measured” in about twenty minutes, and if you are working through an on-prem hardware decision, this is the measurement that changes the purchase order.

What to publish, and the number that gets forwarded

The artifact is small. Two plots, your measured ridge point, and four sentences explaining the shape of the throughput curve.

The last row is the one that gets forwarded. Take your peak sustained output throughput, multiply out to a million tokens, and divide by what the box costs to run for that long including power. Now you have a cost per million tokens for your hardware and your model, which you can put next to a hosted API price list. That number is what turns an LLM serving benchmark into a business argument, and it is the number missing from almost every writeup on the internet.

Closing the series: ten parts, one plot

Every earlier part of this series contributes something to reading your own benchmark output.

• Part 1, the complete inference path: what a single request actually consists of, so you know what you are timing.
• Part 2, byte-pair encoding: tokens are the unit your benchmark counts, and tokens per second means nothing without knowing whose tokenizer.
• Part 3, inside one transformer block: the seven matrices, one of which is the exact (B, 8192) @ (8192, 8192) shape script 01 measures.
• Part 4, multi-head attention: where the KV cache comes from, which is what runs out first.
• Part 5, the output projection: another square projection with the same shape and the same memory behaviour.
• Part 6, the feed-forward network: where most of the weight bytes live, so most of what your decode step is reading.
• Part 7, mixture of experts: why an MoE model lands somewhere completely different on the same roofline plot.
• Part 8, the 2017 paper dissected: the reference figure, so you can map any architecture variant back to the shapes you are measuring.
• Part 9, the roofline model: the theory that script 01 turns into a number on your own card.
• Part 10, continuous batching and PagedAttention: the scheduler that produces the flat TPOT region, and the preemption warnings that explain where it ends.

Ten parts of theory. One afternoon of measurement. That is the honest way to benchmark LLM inference: the theory tells you what shape to expect, and the measurement tells you where your box actually stops, which is the only number you can build a plan on. If you are moving something from prototype to real traffic, that number belongs in the production readiness checklist before anything else.

Run it tonight. One LLM serving benchmark on your own hardware will teach you more by midnight than six months of reading other people’s charts.

Key takeaways

• Take two measurements. The hardware roofline and the serving sweep. The gap between them is the finding.
• Sweeping batch size sweeps arithmetic intensity, because the weight matrix dominates the bytes moved. That makes the decode-batching argument measurable rather than assertable.
• Expect 10 to 20 percent below datasheet on bandwidth and peak. That shortfall is your real number, and it is what you should quote.
• TTFT, TPOT, end-to-end latency and throughput are four different things. Report p50 and p99 for the latency ones, never the mean.
• Flat TPOT means memory-bound and extra users are nearly free. Rising TPOT means you hit a wall, and only the server log tells you which one.
• p99 TTFT pulling away from p50 is queueing. Fix it with admission control. A bigger card will not help.
• Any LLM serving benchmark reported without model, quantisation, context length and server flags is not a measurement.

Frequently asked questions

What is the difference between TTFT and TPOT?

TTFT is the wall clock time from sending the request to receiving the first token, and it is dominated by prefill compute and by how long you waited in the queue. TPOT is the average gap between consecutive tokens after the first, and it is dominated by memory bandwidth and by how many sequences are decoding alongside you. They respond to different knobs, which is why a single latency number is useless.

Why is my measured memory bandwidth lower than the datasheet says?

A 10 to 20 percent shortfall is normal and expected. Datasheet bandwidth is a theoretical peak that assumes no refresh overhead, no ECC, and a perfectly streaming access pattern. Your measured figure is the one your model will actually get, so it is the one worth quoting.

How do I tell whether rising latency is a KV cache problem or a compute problem?

Read the server log at the concurrency level where TPOT started climbing. Preemption or swap warnings mean you exhausted the KV cache, which is a memory problem fixed with shorter context, more cache headroom or KV quantisation. A clean log with GPU utilisation near 100 percent means you reached the compute knee, which is not a problem at all.

Should I use tensor parallelism or run two independent replicas?

Measure it, because the answer depends on your interconnect. Without NVLink, tensor parallelism pays an all-reduce cost across PCIe on every layer, so two independent replicas behind a load balancer usually win on total throughput. Tensor parallelism still wins when the model does not fit on one card, or when single-request latency matters more than aggregate throughput.

Does prefix caching make my benchmark results look better than they are?

Yes, if your test prompts share a prefix and your real traffic does not. Randomise the prompts so prefill work is genuinely repeated, then run a second sweep with a deliberately shared prefix if that reflects your production workload. Reporting both is more honest than reporting either alone.

What concurrency limit should I actually set in production?

Set it at the point where TPOT stops being flat, which is usually well below the point of peak throughput. Past the flat region every additional admitted request buys queueing delay that your users feel and buys very little extra throughput in return.

Sources and further reading

• Roofline: An Insightful Visual Performance Model for Multicore Architectures, Williams, Waterman and Patterson, the original model that script 01 measures.
• Efficient Memory Management for Large Language Model Serving with PagedAttention, the vLLM paper, including the preemption and swapping behaviour you will see in the logs.
• vLLM documentation, for the current set of server flags, sampling parameters and the OpenAI-compatible API surface.
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, on why KV cache size is the constraint that usually binds first.
• FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, on why prefill behaves so differently from decode.
• torch.cuda.Event, the timing primitive used in the roofline probe.