AdamW Explained, Line by Line

See where AdamW runs inside the training loop

Start with placement, because it makes everything after it concrete. A training step has four stages, and they repeat for every batch of examples.

1. Forward. Push a batch of text through the model and get predictions out.
2. Loss. Compare those predictions against the text that actually came next. The result is one number saying how wrong the model was.
3. Backward. Work out, for every single weight, which direction that weight should move to make the loss smaller. That is one number per weight, and it is called the gradient.
4. Optimizer step. Actually move the weights.

AdamW is stage four. Nothing else in that list changes a single weight. In PyTorch the last two stages are literally loss.backward(), then optimizer.step(), then optimizer.zero_grad() to clear the gradients before the next batch.

It is worth being clear about how little AdamW knows. It never sees your text. It does not know what a token is, how many layers the model has, or which weight belongs to which layer. It receives a long list of numbers, one gradient per weight, and returns a decision about how far each of those weights moves. Which positions in your text were graded in the first place is decided long before this, and the next part, on masking, is about that choice.

One more thing about stage four matters for the rest of the series. AdamW keeps a small record for every weight, and that record survives from one step to the next. It has to, because the whole design rests on remembering. That stored record is the largest single consumer of memory in a training run, which is why the part on the four memory tenants gives it 12 of its 16 bytes per parameter.

Read the name: adaptive moment estimation plus weight decay

The name is not a brand. It is a description, and reading it letter by letter tells you most of the design.

Take “moment” first, because it is the only genuinely unfamiliar word. It is a term from statistics. The first moment of a set of numbers is their plain average. The second moment is the average of their squares. Adam keeps both, for every weight, over that weight’s own gradients:

• The first moment, written m, is a running average of the gradient. It says which way this weight has been heading lately.
• The second moment, written v, is a running average of the gradient squared. Squaring removes the minus signs, so v only tracks size. It says how big this weight’s pushes have been lately.

“Estimation” is in the name because Adam never has the true averages. It has cheap running approximations of them, updated a little at a time. “Adaptive” is there because those two numbers let every weight get its own step size instead of sharing one.

The W was added later. It stands for weight decay, and it names a decision about where one small subtraction goes. The last section of this article is entirely about it.

What each ancestor fixed

AdamW is not one invention. It is the end of a chain, where each link fixed one specific complaint about the link before it.

1. Plain gradient descent. Move every weight a fixed fraction of its gradient. Simple, and it remembers nothing at all.
2. Momentum. Keep a running average of recent gradients and move along that instead. The path stops zig-zagging on noisy batches.
3. AdaGrad. Give each weight its own step size, by dividing its step by the running total of its past gradient sizes. Good idea, one flaw: a total only ever grows, so every step size shrank toward zero and long runs ground to a halt.
4. RMSProp. Replace that growing total with a running average that forgets old gradients. Now a weight’s step size can recover instead of starving.
5. Adam. Use both at once: momentum for the direction, RMSProp’s average for the size, plus a small fix for the first few steps.
6. AdamW. Move the weight decay out of the gradient and apply it to the weight directly.

Every one of those patches is still visible in the four lines below. Nothing was thrown away.

Read the AdamW update, one line at a time

Here is the whole algorithm. It is four lines, and AdamW runs them separately for every weight in the model. Before the lines, the cast.

• theta is one weight. One number, such as 0.5. Part 1 used theta for all 1.54 billion parameters at once. Here it means a single one of them, because AdamW does this same arithmetic on each weight on its own.
• g is this weight’s gradient on this step, handed over by the backward pass. Also one number.
• t is the step number. It is 1 on the first step, 2 on the second, and so on.
• lr is the learning rate, the single overall step size you set in the config.
• m and v are the two running averages the previous part introduced. Both start at 0 before the first step, because there is no history yet.

In all four lines the equals sign means “replace the old value with this”. It is an instruction, not a statement of fact.

Line 1: the smoothed direction

m = beta1 * m + (1 - beta1) * g

Every symbol, in words:

• m on the right is the old running average, carried over from the last step. m on the left is the new one.
• beta1 is a setting, 0.9 by default. It is the share of the old average you keep.
• (1 - beta1) is therefore 0.1, the share of the new gradient you mix in.
• g is this weight’s gradient right now.

Read as a sentence: keep 90 percent of what you already had, and stir in 10 percent of the latest gradient. One odd batch can only move m by a tenth of the way, so a single bad example cannot swing the direction. m is the direction, with the noise taken out.

Line 2: the record of how big this weight’s gradients have been

v = beta2 * v + (1 - beta2) * g * g

Every symbol, in words:

• v on the right is the old value, v on the left the new one. Same shape as line 1.
• beta2 is a setting, 0.999 by default. So you keep 99.9 percent of the old value and mix in 0.1 percent of the new one. Much slower to move than m.
• g * g is the gradient multiplied by itself.

That multiplication is the only real difference from line 1, and it does one useful thing: it throws the minus sign away. A gradient of -0.2 and a gradient of +0.2 both contribute 0.04. So v knows nothing about direction. It only knows size, averaged over a long stretch of recent steps.

Line 3: the startup fix

m_hat = m / (1 - beta1 ** t)

v_hat = v / (1 - beta2 ** t)

Every symbol, in words:

• m_hat is said “m hat”. It is the corrected version of m. Same for v_hat.
• beta1 ** t means beta1 raised to the power t, which is beta1 multiplied by itself t times. On step 3 with beta1 at 0.9 that is 0.9 times 0.9 times 0.9, which is 0.729.
• t is the step number again.

Both m and v started at 0. That zero is a lie about the model, and it drags both averages down for the first stretch of the run. These two divisions cancel that lie. The next section but one shows exactly what they are worth at step 1, step 2 and step 100, and how quickly they stop mattering.

Line 4: the step that actually moves the weight

theta = theta - lr * m_hat / (sqrt(v_hat) + epsilon) - lr * lambda * theta

Every symbol, in words:

• theta on the right is the weight’s current value, and on the left its new value.
• lr is the learning rate. It scales the whole move.
• m_hat is the corrected smoothed direction from line 1. This is the “which way” half.
• sqrt(v_hat) is the square root of the corrected size record. The square root undoes the squaring in line 2, so this number is back in the same units as a gradient. This is the “how big have this weight’s pushes been” half.
• epsilon is a tiny number, 1e-8 by default, added so the division can never be a division by zero.
• lambda is the weight decay strength, typically 0.01. It is the only symbol in the line that has nothing to do with the gradient.

The line has two subtractions, and they are separate ideas bolted together. The first subtraction is the Adam part: direction divided by size. The second, lr * lambda * theta, is the W. You can set lambda to 0 and the second subtraction disappears entirely.

Now read that first fraction on its own, because it is the whole point of Adam. If this weight’s gradients have been large, sqrt(v_hat) is large, so the fraction is small and the weight barely moves. If they have been small and consistent, the fraction is close to 1 and the weight moves by roughly the full learning rate. Each weight is measured against its own history, never against the rest of the model.

Follow one weight through three steps by hand

Now watch it run. One weight, small numbers, nothing hidden.

Set the weight theta to 0.500. Learning rate 0.001. The defaults for the rest: beta1 0.9, beta2 0.999, epsilon 1e-8. Weight decay is 0 for now, so the last term drops out. Say the backward pass hands this weight the same gradient, 0.10, on every step.

Step 1, one line at a time:

1. m = 0.9 * 0 + 0.1 * 0.10 = 0.010
2. v = 0.999 * 0 + 0.001 * 0.10 * 0.10 = 0.00001
3. m_hat = 0.010 / (1 - 0.9) = 0.010 / 0.1 = 0.10
4. v_hat = 0.00001 / (1 - 0.999) = 0.00001 / 0.001 = 0.01
5. The square root of 0.01 is 0.1, so the fraction is 0.10 / 0.1 = 1.0
6. theta = 0.500 - 0.001 * 1.0 = 0.499

Two things in that arithmetic are worth stopping on. First, m_hat came out at 0.10, which is exactly the gradient, and v_hat came out at 0.01, which is exactly the gradient squared. From a single sample, the correction handed back the true values. Second, the weight moved by 0.001, which is exactly the learning rate.

Keep going with the same gradient and the pattern holds.

Look at the raw m and v columns crawling upward, and then at the corrected columns sitting perfectly still. That gap is the entire job of line 3.

The result in the last column is the single most useful fact about AdamW. When a weight’s gradients are steady, it moves by about the learning rate every step. Not by the gradient. By the learning rate. Feed the same weight a steady gradient of 100 instead of 0.10 and it still moves by 0.001 per step, because 100 divided by 100 is also 1. Adam divides the size out.

That is why the learning rate is the dial you tune and the gradient scale is mostly not your problem. It is also why a learning rate that is ten times too large is such a reliable way to wreck a run: every weight in the model then travels ten times too far, every step.

Here is the same arithmetic as code. It needs nothing installed.

beta1, beta2, eps = 0.9, 0.999, 1e-8
lr, decay = 0.001, 0.0

theta = 0.5          # the weight
m = v = 0.0          # both running averages start at zero

for t in (1, 2, 3):
    g = 0.10                                       # the same gradient every step
    m = beta1 * m + (1 - beta1) * g                # line 1
    v = beta2 * v + (1 - beta2) * g * g            # line 2
    m_hat = m / (1 - beta1 ** t)                   # line 3
    v_hat = v / (1 - beta2 ** t)                   # line 3
    theta = theta - lr * m_hat / (v_hat ** 0.5 + eps)   # line 4, the Adam part
    theta = theta - lr * decay * theta                  # line 4, the W part
    print(f"step {t}  m={m:.4f}  v={v:.8f}  theta={theta:.4f}")

Change g to 100 and watch the weight land on the same three values. Change decay to 0.01 and watch the last line start shaving a little off the weight as well.

Watch bias correction fade to nothing

Line 3 is the line people skip, so here it is on its own, in numbers.

The problem is the starting value. Both m and v have to begin somewhere before any gradient exists, and zero is the only honest choice. But zero is also a claim: it says this weight has had no gradients and no movement. On step 1 that false claim is 90 percent of m and 99.9 percent of v.

Work out how far off they are. On step 1, m holds 0.1 times the gradient, so it reads at a tenth of the truth. v holds 0.001 times the gradient squared, so its square root reads at about a thirty-second of the gradient’s size. Both are too low, and they are too low by very different amounts.

Now divide them, uncorrected, as line 4 would: 0.1 divided by 0.0316 is about 3.2. Without the correction, the first step would be about three times the learning rate rather than one times it. That is the real damage. The two estimates start cold, they warm up at different speeds, and the ratio between them comes out wrong until they do.

The fix is one division per average, by 1 - beta ** t. Here is what that division is worth as the run goes on. Each column is the number the average gets multiplied by, once you turn the division around.

Read the last column down and the mechanism is obvious. The correction is huge when the averages are empty, and it puts itself out of business as they fill. Nobody switches it off, because beta ** t shrinks toward zero on its own and the division quietly becomes a division by 1.

Two practical notes follow from that table. The v column fades far more slowly than the m column, because beta2 is set to remember far more history. And there is nothing here for you to configure. Bias correction is arithmetic inside the optimizer, not a setting.

It is also a different thing from learning-rate warmup, which is a setting, and one you will meet in every training config. Warmup ramps the learning rate up from zero over the first few percent of steps, to protect the model from big early moves. Bias correction repairs the optimizer’s own two estimates. You want both, and neither replaces the other.

Explain why a jumpy weight takes smaller steps than a steady one

The last two sections used a weight whose gradient never changed. Real gradients change constantly, so put two weights side by side.

Weight A gets a gradient of +0.10 on every step. It is being pushed steadily in one direction.

Weight B gets +0.10, then -0.10, then +0.10, then -0.10. Same size every time, opposite direction every time. Successive batches disagree about which way it should go.

Start with v, because it is the boring one. Line 2 squares the gradient, and squaring kills the sign, so both weights build exactly the same v and exactly the same brake. All the difference lands in m.

Check row 2 by hand if you like: 0.9 * 0.010 + 0.1 * (-0.10) = 0.009 - 0.010 = -0.001. The new gradient cancelled almost everything the first one built.

After step 4, correct both averages by the same factor and divide by the same brake. Weight A’s fraction is 1.0, so it moves by a full learning rate. Weight B’s is about 0.05, so it moves by a twentieth of one, and in the opposite direction to its last move. Keep going and B keeps hovering there, taking tiny steps in alternating directions.

That is the behaviour worth remembering. A weight that keeps changing its mind moves slowly. A weight that gets pushed the same way every step moves at the full rate. Neither one needed a setting for that, and neither weight knows anything about the other. The previous part covers why this matters so much for a transformer, where gradients differ in size by orders of magnitude between the embeddings, the attention layers and the normalisation layers.

Set the learning rate, the two betas and epsilon with intent

Every symbol in line 4 that is not a gradient is a line in your config. Here they all are.

The two betas have a rule of thumb that makes them intuitive. A running average with decay rate beta remembers roughly the last 1 / (1 - beta) steps. That is all there is to it.

• beta1 at 0.9 gives 1 / 0.1, so the direction remembers about the last 10 gradients. Lower it and the optimizer reacts faster and gets noisier.
• beta2 at 0.999 gives 1 / 0.001, so the brake remembers about the last 1,000. That is deliberately much longer. You want the brake to reflect a settled view of this weight’s usual gradient size, not whatever the last batch happened to contain.

The one place the default moves is very large pretraining runs, where 0.95 is common. That is a memory of about 20 steps rather than 1,000. Early in a pretraining run the loss landscape is still changing shape quickly, and a brake built from 1,000 stale steps can be badly wrong about the present. Fine-tuning starts from a model that has already settled and moves it a short distance, so 0.999 is right and there is no reason to touch it.

Epsilon looks like a rounding detail and mostly is one. It has two jobs. The first is protection: if a weight’s gradients have all been zero, v is zero, its square root is zero, and dividing by zero produces infinity or a NaN that kills the run. With epsilon the denominator is 1e-8 instead, and since m is zero too, the step is zero, which is exactly right for a weight with no signal. The second job is damping. Any weight whose sqrt(v_hat) is not much larger than epsilon gets a smaller step than it otherwise would. At 1e-8 that affects almost nothing. Raise it to 1e-6 and it quietly steadies the weights with the tiniest gradients, which is why you sometimes see 1e-6 in a heavily quantised or low-precision recipe that otherwise looks standard.

Tell weight decay apart from an L2 penalty and from gradient clipping

Now the W, which is the last term in line 4 and the reason the letter is there at all.

What weight decay is

Weight decay is a small pull on every weight, toward zero, applied on every step. That is the whole idea. Look again at the term: theta = theta - lr * lambda * theta. With a learning rate of 0.001 and lambda of 0.01, that shrinks every weight by 0.001 percent per step. A weight receiving no gradient at all would end a 3,000-step run about 3 percent smaller than it started.

Why bother shrinking weights on purpose? Because of a failure called overfitting: the model fits your training examples so closely that it does worse on anything it has not seen. Large weights are one way that happens, since they let the model lean very hard on a few specific patterns. A constant gentle pull toward zero means a weight only stays large if the gradients keep insisting on it.

Any deliberate constraint of that kind, added to make a model behave better on unseen data rather than to make the training loss smaller, is called regularisation. Weight decay is the mildest and most common one. Nobody tunes it much. 0.01 is a fine default, 0.0 is a legitimate choice, and above 0.1 is unusual.

Why folding the decay into the gradient goes wrong

There is an older way of asking for the same thing, called an L2 penalty. You add a term to the loss that is proportional to the square of each weight, so a large weight literally costs you loss. Work out its gradient and it comes to lambda times the weight, which means the whole scheme reduces to one instruction: before the optimizer sees g, add lambda * theta to it.

With plain gradient descent those two routes are identical. With Adam they are not, and that is the entire finding of the paper that added the W.

Follow the decay once it is inside g. It goes into m, into v, and then through the division by sqrt(v_hat). So each weight’s decay gets divided by that weight’s own brake. Take two weights, both sitting at 0.5, with a learning rate of 0.001 and lambda of 0.01. One is busy, with a large brake. One is quiet, with a small one. Once the running averages have settled, the decay part of their steps looks like this.

The right-hand column is what you asked for: the same percentage shrink for every weight. The column beside it is what the old route delivers. The quiet weight gets pulled ten times harder than the busy one, for no reason anybody chose. Set weight_decay above zero on plain Adam and this is what you get.

AdamW takes the decay out of the gradient and applies it to the weight directly, after the adaptive part is finished. It costs no extra memory, because it changes where a subtraction happens and stores nothing new. When weight_decay is exactly 0 the two are the same optimizer. That is the whole of the W, and it is why every current recipe says adamw rather than adam.

Weight decay is not gradient clipping

These two get mixed up constantly, probably because they sit next to each other in every config file. They are unrelated.

Gradient clipping works like this. After the backward pass and before the optimizer runs, add up the size of every gradient in the model into one number, called the gradient norm. If that number is above a limit, scale all the gradients down together until it sits exactly at the limit. The standard limit is max_grad_norm = 1.0, and it needs no tuning.

The technique is Pascanu et al.’s. They were working on recurrent networks, where a single batch could produce a gradient large enough to throw the weights somewhere useless in one step, and rescaling the whole gradient back to a fixed norm was the remedy that kept the direction while capping the distance. Transformers are far better behaved, which is exactly why clipping at 1.0 almost never fires and costs almost nothing to leave on.

Clipping costs nothing and prevents the classic mid-training loss explosion, so leave it on. It also gives you a free diagnostic: if the gradient norm is being clipped on almost every step, your learning rate is too hot. The part on running a supervised fine-tune end to end covers how to read that log while a run is in progress.

Choose between AdamW and its alternatives

AdamW has one well-known cost, and it is memory. Two extra numbers per weight, stored at 4 bytes each, and they exist for the whole run. Almost every alternative in practical use is an attempt to pay less of that.

For a fine-tune the decision is nearly always made for you. Start on AdamW. If the run does not fit, move to the 8-bit version, which is the same algorithm with smaller storage and behaviour close enough that you do not have to re-tune anything. Only reach past that once you have hit a wall those two cannot clear, and on a fine-tune you rarely will.

One name is missing from that table on purpose, because it is the alternative that is not about memory at all. You et al.’s LAMB stores as much as AdamW does. It adds a per-layer rescaling on top of the per-weight one, so that training stays stable at batch sizes in the tens of thousands. That is a pretraining problem. A fine-tune on one card never gets near those batch sizes, so it is background rather than a choice.

Three other names turn up in 2026 papers: Muon, Shampoo and SOAP. They aim at frontier-scale pretraining runs rather than at fine-tunes, and they are further reading rather than a decision anyone owes you.

Configure AdamW for a real fine-tuning run

In a training config, all of the above is one string and a handful of numbers.

# TRL SFTConfig or Transformers TrainingArguments. Names current as of August 2026.
optim = "adamw_torch"              # TrainingArguments default; SFTConfig defaults to adamw_torch_fused
# optim = "adamw_torch_fused"      # identical maths, one faster kernel, no memory cost
# optim = "paged_adamw_8bit"       # 8-bit state plus paging, for QLoRA or a tight card

learning_rate     = 2e-4           # LoRA regime; a full fine-tune wants 1e-5 to 2e-5
adam_beta1        = 0.9            # direction memory, about 10 steps
adam_beta2        = 0.999          # brake memory, about 1,000 steps
adam_epsilon      = 1e-8           # floor under the divide in line 4
weight_decay      = 0.01           # the W, applied straight to the weights
max_grad_norm     = 1.0            # gradient clipping, a separate safety limit
lr_scheduler_type = "cosine"       # schedules the learning rate, separate from AdamW
warmup_ratio      = 0.03           # also separate from bias correction

Library APIs drift. Pin your versions and check the current trainer documentation before a real run. The names above are current as of August 2026.

Two picks cover most cases. For a small full fine-tune on one card, the fused variant is the fastest and costs no extra memory, because it runs the same arithmetic in fewer trips to memory. For LoRA or QLoRA on a card that is already close to full, paged_adamw_8bit keeps AdamW’s behaviour, cuts the stored state, and survives a memory spike instead of crashing on it.

What those four lines cost on the running example

Qwen2.5-1.5B has 1.54 billion weights, so AdamW runs its four lines 1.54 billion times per step. Count the storage.

• m and v are one 32-bit number each per weight, so 4 plus 4 bytes. That is 1.54e9 x 8 = 12.3 GB for the two running averages alone.
• A mixed-precision run also keeps a 32-bit master copy of each weight, which the optimizer is the component that updates. Another 4 bytes, so the optimizer tenant comes to 12 bytes per parameter, about 18.5 GB.
• Add the weights and the gradients at 2 bytes each and you have the 16 bytes per parameter this series quotes everywhere: about 24.6 GB of static state for a full fine-tune of this model.

That 24.6 GB is static state only, meaning weights, gradients and optimizer state. Activations sit on top and are counted separately. At a batch of 4 sequences of 1,024 tokens they come to roughly 8 GB, which is an order of magnitude rather than a measurement. So the run wants about 33 GB, against roughly 29.8 GiB usable on a 32 GB card once the driver has taken its share. It does not fit, and that is before leaving the 2 to 3 GB of safety margin every estimate in this series keeps.

Switching to 8-bit takes the m and v part from 12.3 GB to about 3.1 GB, so the optimizer tenant drops from 12 bytes per parameter to about 6. LoRA attacks the same bill from the other end: it freezes almost every weight, and a frozen weight gets no gradient, no m and no v. Static state for this model at rank 16 falls to about 4 GB. The same roughly 8 GB of activations is still owed on top of that, because freezing weights does not change the forward pass.