Training Memory: The Four Tenants and the 16 Bytes Per Parameter

Training memory is the reason a model that occupies 3 GB on disk needs 25 GB or more of VRAM to fine-tune. The gap is not waste. It is the bookkeeping a training step needs in order to compute a gradient and apply an update, and every byte of it has a name and a job.

This part derives the famous figure, 16 bytes per parameter, from first principles. By the end you will be able to itemise where each of those bytes goes, explain why the model’s weights appear twice in the accounting, and say exactly which of the four costs LoRA removes and which it leaves untouched.

The previous part established the one-line cost model: memory is bytes per parameter times parameters, and the overview of what fine-tuning actually changes used 16 without deriving it. Here is the derivation.

What training memory actually means

Training memory is the total GPU memory a fine-tuning run occupies while it is running. The crucial fact is that it is several times larger than the parameter count suggests, and the excess exists only to answer one question: how should each weight change?

Inference needs almost none of that machinery. It runs the model forward and reads the output. That asymmetry is why the same model is cheap to serve and expensive to train.

This is also why sizing a GPU by parameter count is the wrong habit. Parameter count tells you the size of exactly one of the four things you have to hold. To plan a run you multiply parameters by 16 for standard mixed-precision AdamW, then add activation memory, which does not scale with parameters at all. The question is never “is 7 billion times 2 bytes small enough”. It is “what is the per-parameter rate for the workload I am running”.

The four stacked tenants

Four distinct kinds of data occupy the card during training. Three of them scale with parameter count and together make the 16 bytes. The fourth scales with batch size and sequence length instead.

Taking them in order:

• Weights, 2 bytes per parameter. The bf16 copy the forward and backward passes actually read.
• Gradients, 2 bytes per parameter. One number per weight, in bf16, saying which way and how hard to nudge that weight. Exactly one gradient per weight, which is why this tenant is the same size as the weights tenant.
• Optimizer state, 12 bytes per parameter. A 32-bit master copy of the weight plus AdamW’s two running statistics, all in fp32. This is three quarters of the bill and the subject of most of this part.
• Activations, sized by batch times sequence times hidden times layers. Every intermediate tensor the forward pass produced that the backward pass will need. The next part follows one activation from birth to death and puts numbers on it.

Stacked means they cannot take turns

The obvious optimisation is to compute the gradients, free the activations, then run the optimizer, reusing the same memory. It does not work, and the reason is a dependency chain. The optimizer step needs the gradients. The gradients were produced using the activations. The whole set is live from the end of the forward pass through to the update.

So peak memory is the sum, not the maximum, and the sum is what has to fit. Put the running example through it. Qwen2.5-1.5B has about 1.54 billion parameters, so 16 times 1.54e9 is roughly 24.6 GB for the three parameter-scaled tenants. Add about 8 GB of activations at batch 4 and sequence 1024 and the peak lands near 33 GB. On a card advertised as 32 GB, which gives about 29.8 GiB of usable VRAM, that does not fit without the memory tricks in Part 9.

That single multiplication, 16 times parameters, is the number that tells you a 1.5B full fine-tune is tight and a 7B full fine-tune is impossible on one card. Every efficiency technique in this series exists because of it.

Mixed precision: two copies, two jobs

Look at the tenant list again and you will notice the weights appear twice: a 2-byte bf16 copy and a 4-byte fp32 copy inside the optimizer. That duplication is not sloppiness. It is mixed-precision training, and it is how every modern run works.

The idea: do the heavy arithmetic, forward and backward, in 16-bit for speed and half the memory, but keep a full 32-bit master copy of the weights so that tiny updates do not get rounded away when applied. You get 16-bit throughput and 32-bit precision, each where it matters.

The precision argument is concrete. Near 0.5, bf16’s 7 mantissa bits give a gap between representable values of about 0.0039. An update of 0.0001 is far below that, so 0.5000 plus 0.0001 rounds straight back to 0.5000 and the update is gone. fp32’s 23 mantissa bits give a gap near 0.5 of about 6e-8, which captures that update easily. Across thousands of steps the difference is training that progresses versus training that stalls. Part 4 works through the number formats in detail, including why this failure mode has a name.

One historical note that explains a config line you will see everywhere. Classic mixed precision used fp16 and needed loss scaling, which means multiplying the loss by a large constant before backpropagation so that small gradients do not underflow fp16’s narrow range, then dividing them back before the optimizer step. bf16 shares fp32’s exponent range, so gradients do not underflow and loss scaling is usually unnecessary. That is why modern recipes set bf16=True rather than fp16=True on hardware that supports it. The memory accounting, 2 plus 2 plus 12, is identical either way. bf16 just removes a tuning knob and a source of silent NaN values.

AdamW, and why 12 bytes is a fair price

AdamW is the optimizer almost every LLM fine-tune uses, and it is the reason the optimizer tenant dominates. An optimizer’s job is to turn a gradient into an actual weight change. The simplest one, SGD, multiplies the gradient by a learning rate and subtracts, storing nothing extra. AdamW does something more useful and, for transformers, far more reliable: it tracks two running statistics per parameter so that every weight gets its own adaptive step size.

Those two statistics are the memory. Each is one fp32 number per parameter, so 4 plus 4 is 8 bytes. Add the fp32 master weight copy at 4 bytes and the optimizer tenant is 12 bytes per parameter, which is 12 of the 16. Part 6 dissects the update equations line by line; here the point is the memory.

What you buy for the 8 bytes is robustness. A transformer’s gradients vary enormously across its parts: attention against feed-forward, early layers against late, layer norms against projections. With SGD’s single global learning rate, a value that is right for one part is wrong for another. The second running statistic auto-scales each weight’s step to that weight’s own gradient magnitude, so one setting works across the whole network. The field pays 8 bytes per parameter to avoid a brutal tuning problem.

The W is decoupled weight decay

The W distinguishes AdamW from plain Adam. Weight decay is a regulariser that gently pulls weights toward zero to curb overfitting. Original Adam folded it into the gradient as an L2 penalty, where Adam’s per-weight scaling then distorted it, so weights with large second-moment estimates got their decay shrunk and regularisation landed unevenly. AdamW applies the decay directly to the weight, outside the adaptive step, so it acts uniformly as intended.

The fix is free in memory, since it changes the update rule rather than the stored state. That is why every current recipe and every framework default says adamw and not adam, and why setting weight_decay above zero with plain Adam is now treated as a mild bug.

Where the give is

Because the optimizer is three quarters of the bill, it is also where the savings are when a run will not fit.

Eight-bit Adam quantises the two moments and reclaims most of 8 bytes per parameter, keeping the algorithm and its behaviour intact. Paged optimizers offload the state to host RAM during a spike and bring it back, turning a hard crash into a brief slowdown. LoRA removes the tenant entirely for the frozen base. All three are covered later in the series, and all three exist because of the figure above.

A tempting alternative is worth ruling out explicitly. You could switch to SGD and halve the optimizer tenant. In practice transformer training with SGD is notoriously finicky, needs careful learning-rate schedules, and often still trains worse. When memory binds, the better trade is almost always to keep AdamW and shrink its state, or to freeze most of the model, rather than to give up per-weight adaptivity.

The full bill, and where LoRA cuts

Now the four tenants assemble into one picture, and the picture reveals exactly what parameter-efficient fine-tuning does. LoRA does not shrink the model. It removes tenants for the weights it freezes.

Here is the whole stack in one table, including what removes each tenant.

That last row is why the part on LoRA still tells you to enable gradient checkpointing. The forward pass still runs through every frozen layer, and the backward pass still flows all the way back through them to reach the adapters, so every forward activation still has to be stored. Freezing saves you optimizer state and weight gradients. It never saves you activations.

The two questions people tangle

Two questions about this material look like one question and are not, and mixing them up is the single most common error when people first learn the tenant model.

Question one: is this thing a training memory tenant? Question two: is this thing part of the 16 bytes per parameter? Activations answer yes to the first and no to the second, because they do not scale with parameters. The KV cache answers no to both during training, because there is no generation loop while training.

A second look-alike worth separating. The 12-byte optimizer breakdown is the fp32 master copy plus the two running averages, at 4 bytes each. It does not include the bf16 compute weights, which are their own 2-byte tenant, and it does not include the gradients, which are another 2-byte tenant. Three fp32 numbers per parameter, no more and no less.

If you can answer both questions separately and recite the 12-byte breakdown without hesitating, the memory model is yours, and every OOM error you meet from here is diagnosable on sight. The same discipline applies when you reason about whether to move an LLM workload on-prem, where the training and serving footprints have to be budgeted separately.

Key takeaways

• Training memory is the VRAM a run occupies while training, and it is far more than the model’s size because a training step needs the bookkeeping to compute and apply an update.
• The four stacked tenants are weights, gradients, optimizer state and activations. All four are live at the peak, so peak memory is their sum rather than their maximum.
• Three tenants scale with parameters and total 16 bytes each under standard mixed-precision AdamW: weights 2, gradients 2, optimizer 12.
• Mixed precision keeps two weight copies on purpose. bf16 for fast compute, fp32 as a master so that updates smaller than bf16’s resolution are not rounded to nothing.
• The 12 bytes are an fp32 master copy plus AdamW’s two fp32 running averages. They buy per-weight adaptive step sizes, which is what makes transformer training reliable.
• Freezing a weight removes 14 of its 16 bytes, which is the LoRA memory win told from the tenant side. It does not remove activations.
• Being a tenant and being part of the 16 bytes are two different questions. Activations are a tenant and are not in the 16.

Frequently asked questions

Why does a 3 GB model need 25 GB of VRAM to train?

Because the weights are only one of four things resident during a training step. A gradient is stored for every weight, the optimizer stores three more numbers per weight, and every intermediate tensor the forward pass produced is held until the backward pass consumes it. Under standard mixed-precision AdamW the parameter-scaled part alone is 16 bytes per parameter, and activations sit on top.

What exactly are the 16 bytes per parameter?

Two bytes for the bf16 weight copy used in compute, two bytes for the bf16 gradient, and twelve bytes of optimizer state, which is a 4-byte fp32 master copy of the weight plus two 4-byte fp32 running averages of the gradient and the squared gradient. Activations are a real cost but are not part of this per-parameter figure.

Is the KV cache part of training memory?

No. The KV cache exists to avoid recomputing the keys and values of past tokens during token-by-token generation, and training has no generation loop. It is an inference tenant. Confusing the two columns of the tenant map is the most common mistake people make with this material.

Can I just use bf16 everywhere and skip the fp32 master copy?

Not without stalling training. Near 0.5, bf16’s smallest representable step is about 0.0039, so an update of 0.0001 rounds away to nothing and the weight never moves. The fp32 master accumulates those small increments correctly while the bf16 copy handles the fast arithmetic.

Should I switch from AdamW to SGD to save memory?

Usually no. You would save 8 bytes per parameter, but transformer training with SGD needs careful learning-rate schedules and often still ends up worse, because a single global learning rate cannot suit gradients that vary by orders of magnitude across the network. Quantising AdamW’s state to 8 bits or freezing most of the model with LoRA keeps the good behaviour and saves more.

Does LoRA reduce activation memory?

No, and this is the most common LoRA misconception. LoRA removes gradients and optimizer state for the frozen base, which is 14 of the 16 bytes for about 99 percent of parameters. The forward pass still runs through every frozen layer and the backward pass still traverses them to reach the adapters, so the activation bill is unchanged. That is why LoRA runs still enable gradient checkpointing.

Sources and further reading

• Rajbhandari et al., ZeRO: Memory Optimizations Toward Training Trillion Parameter Models, the standard mixed-precision AdamW accounting of 2 plus 2 plus 12.
• Kingma and Ba, Adam: A Method for Stochastic Optimization, the origin of the two running moment estimates.
• Loshchilov and Hutter, Decoupled Weight Decay Regularization, the paper that put the W in AdamW.
• Micikevicius et al., Mixed Precision Training, for the fp32 master weights and loss scaling.
• Dettmers et al., 8-bit Optimizers via Block-wise Quantization, which shrinks the largest tenant.
• Hugging Face, Methods and tools for efficient training on a single GPU.