Supervised Fine-Tuning End to End: The First Real Run

Say what pretraining left behind, and what a fine-tune installs

One idea decides how you read every decision in this part, and it is usually stated backwards.

The common story says a base model has not learned to behave and a fine-tune teaches it. That is not what happens. A model that has read trillions of tokens has already absorbed almost every behaviour you could ask for: polite support replies, terse commit messages, JSON payloads, forum arguments, exam answers. Ask it anything and something fluent comes out. Brown et al. showed the strong version of this with GPT-3, a model given no instruction tuning at all: put a few worked examples in the prompt and it performs tasks nobody trained it on. The capability was already in the weights, and the prompt only selected it. What it lacks is any opinion about which of those behaviours this moment calls for.

So the honest summary is this. Pretraining does not fail to absorb behaviour. It absorbs every behaviour and holds no default policy over them. Supervised fine-tuning does not install a behaviour that was missing. It installs the default: it decides which already-absorbed behaviour surfaces when the prompt does not force one. Wei et al. demonstrated that at scale in FLAN: fine-tune a pretrained model on a large collection of tasks phrased as instructions, and it begins following instructions on task types held out of that collection entirely.

That reframing pays for itself twice. It explains why a thousand carefully written examples can turn a base model into a usable assistant, which is what Zhou and colleagues reported in the LIMA work. And it explains why ten thousand cannot teach it your product catalogue. Allen-Zhu and Li reach the same wall from the other side: they find that whether a fact can be pulled back out of a model is largely settled during pretraining, by how that fact was presented there, and that later training does not rescue one that went in badly. Selecting among absorbed behaviours is cheap. Adding a fact that was never absorbed is not.

What “absorbed” actually means

The word carries a lot of weight, so pin it down. Absorbed knowledge has four properties, and each one has a consequence.

1. It is frequency-weighted. A pattern that appeared constantly presses harder on the weights than one that appeared twice. Nothing is stored once. Things are stored more or less strongly, in proportion to how often the data showed them.
2. It is lossy compression. Trillions of tokens went into a few billion numbers, so nothing is kept word for word. What survives is a tendency: given this context, these continuations are likely. That is how a model can be fluent about a topic and wrong about its details in one sentence.
3. It is not addressable. There is no row to look up and no field to edit. You cannot point at the parameters where “refuse politely” lives, because it does not live in one place. It is smeared across the pile.
4. It depends on conditioning. Which absorbed behaviour comes out is decided by the context in front of the weights. Same model, same numbers, different prompt, genuinely different behaviour. That is why prompting works at all, and why it is unreliable. You are steering rather than setting.

Read those together and fine-tuning stops being mysterious. You cannot edit an unaddressable, lossy, frequency-weighted store directly. You can add a few thousand examples that all press in one direction, and shift which behaviour wins when nothing else is pushing.

The twelve layers, and the six a fine-tune actually moves

“Behaviour” is too coarse a word to plan a project with. Split what a model holds into twelve layers, from the mechanics of words up to what it believes about itself. This is a working taxonomy, not something the architecture enforces: none of the twelve corresponds to one of the model’s 28 stacked layers. It is a checklist for deciding whether a fine-tune is the right tool at all.

Be plain about where the list comes from. The twelve layers below are this series’ own way of organising the question, not a taxonomy you will find in a paper, and nobody else numbers them this way. The organising is ours. The claims each row makes are not, and the paragraphs after the table give the published evidence for them.

Read the last column as a boundary. Layers 1 to 5 are the base model’s competence, bought with millions of dollars of pretraining, and a well-run fine-tune barely disturbs them. Choose your base model for those five, as Part 8 does, because supervised fine-tuning will not repair them. A badly run fine-tune does damage them, and that damage has a name and a section further down.

Two lines of published work stand behind that boundary. Kaplan et al. and then Hoffmann et al. measured what pretraining scale actually buys, as smooth curves in parameters, data and compute, which is the sense in which layers 1 to 5 are paid for rather than taught. On layer 4 specifically, Allen-Zhu and Li put a capacity figure on stored facts, around 2 bits of knowledge per parameter. That is a property of the pretrained model, and no run of a few thousand examples adds to it.

Layers 6 to 11 are what you are buying: format, the shape of the reasoning, tone, role, what to refuse. Each is a selection among behaviours the model already produces, which is why a few thousand examples is enough. That is the LIMA result read row by row: Zhou et al. got a usable assistant out of 1,000 curated examples, and what those examples supplied was format, style and the habit of answering. None of it was knowledge.

Layer 12 needs a warning, because a fine-tune can make it worse while every other layer improves. Supervised fine-tuning shows one gold answer per prompt, and in a normal dataset every gold answer is confident. Train on ten thousand of those and you have taught the model that confidence is the house style, including where it should have admitted ignorance. Nothing in the loss reports it, and it is part of what preference tuning in Part 13 exists to fix.

See why one forward pass can grade a whole answer

Now the mechanics, starting with what surprises people who have used a chatbot but never trained one. You might expect training to work the way generation does, one token at a time, each guess fed back in to produce the next. It does not. Training shows the model the whole correct answer at once.

Picture a sentence on a card: “The cat sat on the mat.” Now imagine a reader who has to guess each word from the words before it, and who is told the true word immediately after every guess, right or wrong. They guess word two from word one. Then word three from the real words one and two, never from whatever they guessed a moment ago. On to the end of the card.

That is teacher forcing, and it is how every language model in this series is trained. The name and the concern both predate transformers: Lamb et al. in Professor Forcing set out the gap it leaves, since a model trained only on true prefixes has never had its own output as input, which is the mismatch that shows up at generation time. The model is never fed its own guesses during training. It is fed the true text and asked, at every position at the same time, what comes next.

Two consequences follow, and both shape every training script you will read.

1. One pass grades the entire example. A 400-token answer is not 400 separate runs. It is one forward pass, meaning one trip through the model from input to output, and it produces a prediction at all 400 positions at once. Nothing lets position 10 peek at position 11 while it does this, because the architecture forbids it. That rule is the causal mask, and a section below covers what happens when it goes missing.
2. The correct answers look exactly like the input. What you hand over as targets is the same run of tokens you fed in, because the token sitting at position 5 is the right answer for the prediction made at position 4.

That second point confuses everybody the first time. You pass in a list called labels that is a copy of the input. The model shifts it by one internally, scores every position with cross-entropy, and hands back a single number. You never write the shift and you never write the scoring.

You do write one thing: which positions get graded. In a chat fine-tune you want the model to learn how to answer, not how to ask, so the user’s words are switched off and only the assistant’s are scored. The switch is a label value of -100, and Part 7 explains the convention and why that particular number. A later section builds the mask by hand.

Take one training step apart, and read its gradient norm

The word “step” is overloaded here, and the overload causes one specific misreading worth killing before it starts.

A training step is not a model layer

Qwen2.5-1.5B has 28 layers stacked one on top of another. Data enters the bottom, passes through all 28, and a prediction comes out of the top. A layer is a place in the model.

A training step is not a place. It is one full turn of the training loop, and the entire model takes part in it: one group of examples going forward through all 28 layers, one loss computed at the end, one backward pass running back through all 28, and one update applied to every parameter at the same moment. Layers are not visited one per step. Every layer is visited twice in every step.

So training for 500 steps does not train some layers more than others, and a log line reading “step 400” says the loop has gone round 400 times. Throughout this part, one step means one optimizer update, which may be assembled from several smaller passes for reasons a later section explains.

The five moves of one training step

With teacher forcing in place, one step is five moves. They are the same five for every model trained by gradient descent, and everything else in a training script is arrangements around them.

1. Forward pass. Push a batch of examples through the model. Out come predictions at every position.
2. Loss. Compare those predictions against the true next tokens, at the positions you chose to grade. One number comes out, and lower is better.
3. Backward pass. For every one of the model’s 1.54 billion numbers, work out which way it would have to move to make that one number smaller. Those directions are the gradients, one per number.
4. Optimizer step. Move every number a little in the direction its gradient points. How far is a separate decision, and a later section is about it.
5. Zero the gradients. Wipe the slate. This matters more than it looks: the framework adds each new gradient to whatever is already there rather than replacing it. Skip the wipe and the next step is polluted by this one. That same adding behaviour turns out to be useful on purpose, and a later section uses it.

One gradient per parameter, and one number that summarises them all

Move three deserves to be stated precisely, because the rest of this section is arithmetic on it.

The model has N parameters, and here N is about 1,540,000,000. Call them w_1 up to w_N. The loss is one number that depends on all of them. The backward pass produces one number per parameter, answering a single question: if I nudge this one parameter and hold every other still, at what rate does the loss change? That quantity is a partial derivative.

$$g_i=\frac{\partial L}{\partial w_i}$$

• g_i is the gradient of parameter number i. One ordinary number.
• L is the loss for this step, the single how-wrong-were-we score.
• w_i is parameter number i, one of the 1.54 billion.
• The curly d is the partial derivative sign. It reads as “the rate at which the thing on top changes when only the thing underneath moves”. Part 5 builds that from a ball rolling into a valley.

One backward pass produces 1.54 billion of these, one per parameter. Together they are the gradient, a collection exactly the size of the model itself, which is why Part 2 charges 2 bytes per parameter for it.

Nobody can read 1.54 billion numbers per step, so training logs one summary: the length of that collection treated as a single very long arrow. Length from components is the Pythagoras rule, extended from two numbers to 1.54 billion. Square every component, add them up, take the square root.

$${\parallel g\parallel }_2=\sqrt{\sum _{i=1}^N{g}_i^2}$$

• The double bars mean length, and the small 2 says which kind: the ordinary straight-line one, the same measure Pythagoras gives for two numbers.
• The tall sign is a sum. Add up what follows it, for i running from 1 to N.
• g_i squared is that parameter’s gradient multiplied by itself, which makes it positive so gradients pointing opposite ways cannot cancel out.
• N is the number of parameters, 1.54 billion here.
• The square root undoes the squaring, so the answer returns in the same units as one gradient.

Check it on two numbers. If the only gradients were 3 and 4, the sum of squares is 9 plus 16, which is 25, and the square root of 25 is 5. That is the whole calculation, done 1.54 billion terms wide.

Now the point people miss. That result is one scalar for the entire step. Not one per layer. Not one per parameter. One number standing for all 1.54 billion gradients at once, logged as grad_norm. It cannot tell you which layer moved or which parameter was large. It tells you how big the update wanted to be, and that diagnoses a surprising number of failures.

Where the gradient goes: two running averages and one cap

Two things consume the backward pass, and they consume different halves of what it produced.

The first is the optimizer, which works on the individual gradients rather than the summary. AdamW keeps two running averages for every parameter, updated each step.

$$m_t={\beta }_1{m}_{t-1}+(1-{\beta }_1)g_t$$
$$v_t={\beta }_2{v}_{t-1}+(1-{\beta }_2)g_t^2$$

• t is the step number, so m_t is this step’s value and m_(t-1) is last step’s.
• m is the first moment: a smoothed average of recent gradients, which gives a steadier direction than any single noisy batch.
• v is the second moment: an average of the squared gradients, measuring how large and erratic this parameter’s history has been.
• beta1 and beta2 are the decay rates, normally 0.9 and 0.999. They set how much of the old average survives.
• g_t is this step’s gradient for this parameter. Both lines run once per parameter, so there is an m and a v for each of the 1.54 billion, which is 8 of the 16 bytes Part 2 counts. Part 6 walks the full update rule line by line.

The second consumer is the clip, and this one does use the norm. Clipping by the norm comes from Pascanu et al., who introduced it to stop one exploding update from destroying a network that had been training perfectly well until that batch. If the norm exceeds a limit you set, called max_norm, the whole gradient is scaled down until its norm equals that limit exactly.

$$g_{\text{clipped}}=g\times min(1,\frac{\text{max\_norm}}{{\parallel g\parallel }_2})$$

• g is the whole collection of gradients and g_clipped is what the optimizer is allowed to see.
• max_norm is the limit, normally 1.0, set once.
• The min picks the smaller of the two, so when the norm is under the limit the multiplier is exactly 1 and nothing happens.
• When the norm is over, every one of the 1.54 billion gradients is multiplied by the same fraction. A norm of 4.0 against a limit of 1.0 quarters all of them. Directions survive and only the overall size is capped.

Order matters. Clipping runs after the backward pass and before the optimizer step, exactly where the script below puts it, so the clipped gradient is what those running averages absorb. The norm is both a number you read and a control that acts.

Turn a conversation into numbers the model can train on

The model has never seen a conversation. It has no idea what a role called “user” is. It reads a flat run of integers. So something has to sit between your data and the model, and that something is the biggest source of bugs in the pipeline.

Here is one training example from the dataset this part uses, and the text it has to become.

# what one row of the dataset looks like
[{"role": "user",      "content": "Name three coffee shops."},
 {"role": "assistant", "content": "Uptime Coffee."}]

# what the chat template renders it into, as plain text, before tokenizing
<|im_start|>system
(a default system line)<|im_end|>
<|im_start|>user
Name three coffee shops.<|im_end|>
<|im_start|>assistant
Uptime Coffee.<|im_end|>

The tokenizer and the chat template, in that order

Two components do this work, and they are easy to confuse.

The tokenizer splits text into tokens and maps each one to an integer. It ships with the model and is specific to it. Token number 9707 means one thing in Qwen’s vocabulary and something else in another model’s, so a tokenizer and a model always have to come from the same checkpoint.

The chat template is a small text template that also ships with the tokenizer. It decides what a list of messages turns into: which markers open and close each turn, where the role name goes, and whether a default system line is inserted at the top. The markers above, the ones that look like <|im_start|>, are special tokens. A special token stands for structure rather than words, and it has its own vocabulary entry like any other token.

This is why you cannot find the answer’s starting point by counting characters in your own text. You did not write the system line or the markers. The template did, and only the template knows how long they came out.

Find the boundary by rendering the conversation twice

The fix is a trick rather than a parser. Make the template render the same conversation two ways and compare the lengths. Four steps.

1. Render the whole thing. Take the full conversation, user turn and assistant turn together, run it through the template and tokenize it. Call the list of integers full. This is what the model reads.
2. Render it again without the answer. Take the same conversation with the assistant’s message removed, and ask the template to add the opening scaffolding for an assistant turn on the end. The flag is add_generation_prompt, and it exists because that is what you do at generation time: set the model up to speak and let it continue. Tokenize that too, and call it prompt.
3. Compare the lengths. Every token in prompt sits at the same index in full, because the second render is a prefix of the first. So the length of prompt is the index where the answer begins. That is the boundary, in token space, with no guessing.
4. Build the labels. Copy full, then overwrite everything before the boundary with -100. What is left graded is exactly the assistant’s answer, including the token that closes the turn.

Grading that closing token is not a detail. It is how the model learns to stop talking, and a mask that trims it off produces a model that answers correctly and then keeps going forever.

Tell the causal mask apart from the label mask, and catch a model that cheats

Two different masks act inside one training step, and they are constantly confused. Part 7 builds both from nothing. Here is the distinction in the form that matters when you are debugging.

One blocks attention, the other blocks scoring

The causal attention mask lives inside self-attention, the step where each position looks at other positions and mixes in what it finds useful. Its rule is one line: the position at index i may attend to positions 1 through i, and may not attend to positions i+1 through N. It is applied inside every attention layer on every pass, and you never set it. It comes with the architecture.

The label mask lives at the very end of the forward pass, at the loss. Its rule is also one line: any position whose label is -100 is skipped when cross-entropy adds up the penalties. You set this one, by hand in Version A below and through a flag in Version B.

A label mask does not block attention flow

Here is the sentence to keep. Marking a prompt position with -100 removes it from the loss and removes nothing from attention. Those prompt tokens are still read, and later positions still attend to them, which is what you want: an answer has to be conditioned on the question.

Now run the same fact the other way, because that is where the bug hides. Labels have no say over what any position may look at. If the causal mask goes missing from the attention path, a response token at position i still attends forward to positions i+1 and beyond, whatever the labels say. Setting a label to -100 does not close an attention window, and setting a label to a real token does not open one.

Carry it as a rule. The label mask controls what is graded. The causal mask controls what is visible. Neither substitutes for the other, and confusing them produces a bug whose every symptom is reassuring.

The cheating model, step by step

Nothing in this part’s scripts can produce this failure, because AutoModelForCausalLM applies the causal mask for you. It appears when someone writes a custom attention block, ports a model by hand, or wires a bidirectional encoder-style path into a next-token objective. Know it on sight, because the loss curve actively lies about it.

1. Training loss collapses toward zero, unnaturally fast. Position i is being asked to predict the token at i+1, and that token is now visible in its own input. Copying beats learning, so the model copies. A loss that should settle near 1.0 after thousands of steps instead falls to a fraction of that within a few hundred.
2. Held-out loss looks perfect too. This is the check that normally saves you, and here it does not. The held-out set runs through the same buggy bidirectional path, so it can copy just as easily. Both curves agree, and both measure the same cheat.
3. The gradient norm stays calm. It does not spike. Copying the next token is an easy optimisation with a smooth surface: the model finds it quickly and the gradients shrink and stay small. So the norm sits stable or drifts toward zero, which reads as a beautifully converged run. The usual advice to watch for a gradient spike will not catch this, because there is no spike to catch.
4. Then inference collapses. At generation time position i+1 does not exist yet, because that token has not been produced. The model has learned a function of information that is no longer there, so it emits gibberish, or loops one phrase forever, or stops immediately. The distance between a flawless training log and a useless model is the whole signature.

Confirming it takes one forward pass. Run a single example, note the logits at some position i, then change the token at position i+1 and run it again. In a correct causal model the logits at position i cannot move. If they do, attention is looking forward and every symptom above follows.

Choose the few numbers that decide whether it learns

The five moves are fixed. What you choose is a short list of settings, called hyperparameters because you set them before training rather than the model learning them. Two of them decide whether the model learns anything useful. The rest are housekeeping.

The learning rate is the master dial

The gradient tells each weight which direction to move. It says nothing about how far. The learning rate decides that, and it multiplies every weight change in the run.

Picture walking down a hillside in thick fog. You can feel which way the ground slopes, so you know the direction. The learning rate is your stride length. Stride too far and you cross the bottom of the valley and end up higher on the other side, then overshoot again, worse each time. Stride too short and you are still on the hillside at nightfall.

Both failures are common and they look different in the logs. Too high shows up as a loss that spikes, oscillates, or turns into NaN, which is what floating-point arithmetic produces when asked something undefined and which kills a run permanently. Too low shows up as a loss that barely moves, which leads people to conclude fine-tuning does not work when really they underpowered it.

Two regimes matter, and mixing them up is one of the three most common beginner mistakes in this field.

• Full fine-tuning of a 1B to 2B model wants roughly 1e-5 to 2e-5. You are moving every pretrained weight, and large moves wreck what pretraining spent millions of dollars building.
• LoRA wants roughly 1e-4 to 3e-4, about ten times more, because only a small added patch is training and it starts from near zero. Part 11 covers LoRA.

Copy a LoRA rate into a full fine-tune and you will destroy the model. Copy a full fine-tune rate into LoRA and it will barely move. For this run, start at 2e-5.

Warm the rate up, then let it decay

Two more settings shape the learning rate over time rather than setting it once.

At step zero the optimizer has no history. AdamW sizes each weight’s step from the two running averages above, and on the first step those averages are empty, so the sizing is a guess. A full-size step on a guess can damage the model before training has begun. Warmup ramps the rate from zero to the target over the first few percent of steps.

Then a schedule brings it down. Cosine decay is the reliable default: a smooth curve from the peak toward zero across the rest of the run, so late training polishes instead of shoving. Set warmup as a fraction of the run rather than a step count, so it still makes sense when the dataset or epoch count changes.

Micro-batches, accumulation, and the batch you actually train with

A batch is a group of examples processed in one go. Bigger batches give a steadier gradient, because averaging over 32 examples cancels the quirks of any single one. So bigger is better, up to the point where memory runs out: every example in a batch has its activations stored until the backward pass consumes them, and Part 3 is about that cost.

Gradient accumulation breaks that ceiling. Run four small batches one after another without stepping. Their gradients add up in place, which is the framework’s default, as move five pointed out. After the fourth, take one optimizer step and wipe.

The result is an update built from 16 examples while only 4 examples of activations were ever in memory. The small batch that fits is the micro-batch. The number that matters for training behaviour is the effective batch:

effective batch = per-device batch x accumulation steps x number of GPUs

Here that is 4 times 4 times 1, so 16. For supervised runs on small models, aim for 16 to 64. The effective batch is the number worth quoting because it is the one the optimizer sees, and it is the number Goyal et al. tie the learning rate to in their linear scaling rule for SGD: multiply the batch by k and multiply the rate by k with it.

Divide the loss by the accumulation count

Accumulation comes with one trap that has quietly ruined a lot of runs, and it is invisible.

Gradients are summed across the four micro-batches, not averaged. So the total handed to the optimizer is four times what one batch would have produced, and the optimizer multiplies that by the learning rate. You have taken a step four times too large while the config still says 2e-5.

The fix is to divide each micro-batch’s loss by the accumulation count before its backward pass. Four quarters add up to one whole, and the sum becomes an average again. Without it, the optimizer sees a gradient four times larger than the one your configuration describes. It is tempting to say that this acts like a learning rate four times higher, and with plain SGD it would. With AdamW it does not, and the reason is worth knowing. AdamW divides the step by the square root of the second moment, so multiplying every gradient by four multiplies the first moment by four and the second moment by sixteen, and the square root of sixteen is four. The two fours cancel. AdamW is very nearly blind to a constant rescaling of the gradient.

What is not blind to it is gradient clipping. max_grad_norm acts on the raw norm, before any of that normalisation, so a gradient four times too large trips the clip on nearly every step. Once the clip is firing constantly, the update is set by the clip rather than by your learning rate, and the run is no longer the one you configured. That is the observable to watch for: not a divergent loss, but clipping on almost every step and a loss that will not settle. A trainer does this division for you. A hand-written loop does not.

Starting values that work

Change one dial at a time and watch the loss curve. The order of impact is the learning rate first, then the effective batch, then the memory settings and only if you ran out of memory. Three clean single-variable runs teach you more than thirty tangled ones.

Make a 1.5B full fine-tune fit on one 32 GB card

Before running anything, check that it fits. Qwen2.5-1.5B has 1.54 billion parameters, a figure from Qwen’s own technical report, and standard mixed-precision AdamW costs 16 bytes per parameter of static state, meaning weights plus gradients plus optimizer state. Part 2 derives all sixteen of those bytes.

1.54e9 x 16 bytes = 24.6 GB of static state

Compare like with like, because card memory is quoted in the binary unit. 24.6 GB is 22.9 GiB, and a 32 GB card offers roughly 29.8 GiB once the driver has taken its share. That leaves about 6.9 GiB, and 2 to 3 GB of it should stay free as a safety margin.

Activations have to fit in what is left. At a batch of 4 and a sequence length of 1024 they run to roughly 8 GB, about 7.5 GiB. Treat that as an order of magnitude rather than a measurement, because the exact figure depends on which intermediate values the framework keeps. Either way it does not fit into 6.9 GiB, so the run needs help. Three settings provide it.

bf16 mixed precision

Do the arithmetic in a 16-bit format for speed and memory, while keeping a 32-bit copy of the weights that the optimizer actually updates. The 32-bit copy exists because a 16-bit number cannot record an update far smaller than itself, so without it thousands of tiny changes would round away. Use bf16 rather than fp16, because fp16’s smaller range makes small gradients vanish and forces a loss-scaling workaround. Part 4 covers the formats.

Gradient checkpointing

Do not store every layer’s activations. Store a few, throw the rest away, and recompute what is missing during the backward pass. You buy a large cut in activation memory for roughly 20 to 30 percent more time, about 20 percent by Hugging Face’s own documentation and about 30 percent in the paper that introduced the technique. This is usually the setting that turns an out-of-memory crash into a working run.

An 8-bit optimizer, only if you are still short

AdamW’s two running averages are normally 4 bytes each. An 8-bit optimizer stores them in 1 byte each, taking the optimizer’s share from 12 bytes per parameter to about 6, since the 32-bit master copy is untouched, and the total from 16 to about 10. Reach for it only when the first two settings have not left enough room.

Run a complete fine-tune from one script (Version A, the bare loop)

Here is the whole thing in one piece. It downloads a model, prepares about 10,000 examples, masks the labels by hand, trains for one epoch and saves the result. Nothing is hidden inside a framework.

Read it once without worrying about individual lines. The next section takes it apart block by block, including the lines that look like decoration and are not.

import torch
from torch.utils.data import DataLoader
from transformers import (AutoModelForCausalLM, AutoTokenizer,
                          get_cosine_schedule_with_warmup)
from datasets import load_dataset

dev, mid = "cuda", "Qwen/Qwen2.5-1.5B"           # the BASE checkpoint, not -Instruct

# 1. the tokenizer and the model
tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.float32)   # explicit fp32
model = model.to(dev)
model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
model.config.use_cache = False                   # caching and recomputation cannot both be on
model.train()

# 2. one example in, input ids and labels out
def build(example, tok, max_len=1024):
    msgs = example["messages"]
    # grade the LAST assistant turn, mask everything before it
    last = max((i for i, m in enumerate(msgs) if m["role"] == "assistant"), default=-1)

    # return_dict=False is required: this call returns a dict of tensors by
    # default, and what this function wants back is a plain list of token ids
    full   = tok.apply_chat_template(msgs, tokenize=True, return_dict=False,
                                     add_generation_prompt=False)
    prompt = tok.apply_chat_template(msgs[:last], tokenize=True, return_dict=False,
                                     add_generation_prompt=True)

    full = full[:max_len]
    labels = list(full)
    for i in range(min(len(prompt), len(full))):
        labels[i] = -100                         # switch off the prompt region
    return {"input_ids": full, "labels": labels}

# 3. the dataset and the batches
ds = load_dataset("HuggingFaceH4/no_robots", split="train").map(
    lambda ex: build(ex, tok),
    remove_columns=["messages", "prompt", "prompt_id", "category"])
ds = ds.filter(lambda r: any(l != -100 for l in r["labels"]))   # drop what truncation ate

def collate(rows):                               # pad to the longest row in the batch
    m = max(len(r["input_ids"]) for r in rows)
    ii = [r["input_ids"] + [tok.pad_token_id] * (m - len(r["input_ids"])) for r in rows]
    am = [[1] * len(r["input_ids"]) + [0] * (m - len(r["input_ids"])) for r in rows]
    lb = [r["labels"]    + [-100]              * (m - len(r["labels"]))    for r in rows]
    return torch.tensor(ii), torch.tensor(am), torch.tensor(lb)

loader = DataLoader(ds, batch_size=4, shuffle=True, collate_fn=collate)

# 4. the optimizer and the schedule
accum, epochs = 4, 1                             # effective batch = 4 x 4 = 16
opt = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
total = (len(loader) // accum) * epochs
sched = get_cosine_schedule_with_warmup(opt, int(0.03 * total), total)

# 5. the loop
for ep in range(epochs):
    for step, (ii, am, lb) in enumerate(loader):
        ii, am, lb = ii.to(dev), am.to(dev), lb.to(dev)
        with torch.autocast("cuda", dtype=torch.bfloat16):
            out = model(input_ids=ii, attention_mask=am, labels=lb)   # 1 FORWARD, 2 LOSS
        (out.loss / accum).backward()                                 # 3 BACKWARD
        if (step + 1) % accum == 0:
            torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
            opt.step(); sched.step(); opt.zero_grad()                 # 4 STEP, 5 ZERO
        if step % 80 == 0:
            print(f"ep{ep} step{step} loss {out.loss.item():.3f}")

model.save_pretrained("qwen15b-sft/final"); tok.save_pretrained("qwen15b-sft/final")

One caution before you run it. These library APIs move between releases, so pin your versions and check the current documentation before a real run. Everything here is current as of August 2026.

Read Version A one block at a time

Five blocks, in the order the script runs them. Every excerpt is lifted verbatim from above.

Block 1: load the tokenizer and the model

tok = AutoTokenizer.from_pretrained(mid)
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.float32)
model = model.to(dev)
model.train()

The two from_pretrained calls download a checkpoint from the model hub, cache it on disk, and build the object. Both are given the same identifier, which is the point: a tokenizer and a model must come from the same checkpoint or the integers mean different things at each end.

The word “CausalLM” in the class name means a model that only ever looks left, predicting each token from the ones before it. That class applies the causal mask discussed above, and attaches the output head that scores every token in the vocabulary, which is what makes a loss computable.

The next line moves the model onto the graphics card. Weights are built in system memory first, and model.to("cuda") copies them across. Everything in one computation has to live on the same device, which is why the loop later moves each batch across too.

Then model.train(), which trains nothing. It flips the model into training mode, changing the behaviour of a few layers that act differently while learning. Its counterpart is model.eval(), and forgetting to switch is a classic source of results that do not reproduce.

Why the number format is written out in full

The argument dtype=torch.float32 says which number format the weights are stored in, and it is the least decorative line in the script.

Leave it off and the loader does not quietly fall back to 32-bit. It reads the format declared in the checkpoint’s own configuration file, and Qwen2.5 declares bf16. So omitting the argument loads 16-bit weights, and nothing warns you.

That single omission changes what you are running. With 16-bit weights there is no 32-bit master copy for the optimizer to update, so the tiny updates a low learning rate produces round away against a weight that cannot represent them. The torch.autocast block further down becomes a no-op, because everything is already 16-bit. You get half the memory and none of the protection, and the run still looks fine while it happens.

With the weights genuinely in 32-bit, autocast casts values to bf16 for each forward pass while the stored weights, the gradients and the optimizer’s two averages stay 32-bit. Count the bytes for one parameter: 4 for the weight, 4 for its gradient, 4 and 4 for the averages. That is the same 16 bytes per parameter as the canonical 2 plus 2 plus 12 split, arranged differently.

Why gradient checkpointing takes a keyword argument

model.gradient_checkpointing_enable(gradient_checkpointing_kwargs={"use_reentrant": False})
model.config.use_cache = False

PyTorch has two implementations of checkpointing. The older one, called reentrant, needs at least one input to each checkpointed block to be carrying a gradient, which stops working the moment part of the model is frozen. The newer one has no such limitation and PyTorch’s documentation recommends it. It is not the default in every code path, so pass it explicitly.

The second line switches off the cache. During generation a model saves intermediate work for every token it has produced so it does not redo it for the next one. Training has no generation loop, so nothing is reused, and the cache conflicts with recomputing activations anyway. Set it yourself so the intent is written down.

Block 2: turn one example into input ids and labels

This is the build function, the boundary trick in code. Three excerpts.

last = max((i for i, m in enumerate(msgs) if m["role"] == "assistant"), default=-1)

Find the position of the last assistant message. The default=-1 stops the line crashing on a malformed row with no assistant turn at all.

full   = tok.apply_chat_template(msgs, tokenize=True, return_dict=False,
                                 add_generation_prompt=False)
prompt = tok.apply_chat_template(msgs[:last], tokenize=True, return_dict=False,
                                 add_generation_prompt=True)

The two renders. The first takes every message and adds no generation scaffolding, because the answer is already there. The second stops before the last assistant message and does add the scaffolding, so it ends exactly where the answer would start.

The argument return_dict=False is not optional. This call now defaults to returning a dictionary of tensors, and the rest of the function wants a plain Python list of integers so it can slice and copy it. Leave the argument off and the following lines will not do what they appear to do.

full = full[:max_len]
labels = list(full)
for i in range(min(len(prompt), len(full))):
    labels[i] = -100

Cut the sequence at the maximum length. Copy it to make the labels, since the targets are the inputs. Then walk from the start to the boundary and switch every position off. The min() guard covers the case where truncation cut into the prompt itself, so the loop never runs past the end of the list.

That is the job the trainer’s assistant_only_loss flag does automatically. Doing it by hand once shows there is no magic in it, and it leaves you a method that works when a template will not cooperate.

Block 3: build the dataset and the batches

ds = load_dataset("HuggingFaceH4/no_robots", split="train").map(
    lambda ex: build(ex, tok),
    remove_columns=["messages", "prompt", "prompt_id", "category"])
ds = ds.filter(lambda r: any(l != -100 for l in r["labels"]))

The first call downloads about 10,000 human-written instruction and response pairs. The .map() runs build over every row. The remove_columns argument throws away the original text columns, which matters: a batch has to end up a rectangle of numbers, and leftover strings break that.

The filter is the interesting line. Some answers sit past the truncation point, so cutting at 1024 tokens removes the graded region and leaves an example whose labels are all -100. It produces no loss and no gradient, still costs a forward pass, and is invisible in the logs. A whole batch of them is worse: there is nothing to average, and the loss comes back as NaN.

def collate(rows):                               # pad to the longest row in the batch
    m = max(len(r["input_ids"]) for r in rows)
    ii = [r["input_ids"] + [tok.pad_token_id] * (m - len(r["input_ids"])) for r in rows]
    am = [[1] * len(r["input_ids"]) + [0] * (m - len(r["input_ids"])) for r in rows]
    lb = [r["labels"]    + [-100]              * (m - len(r["labels"]))    for r in rows]
    return torch.tensor(ii), torch.tensor(am), torch.tensor(lb)

Examples have different lengths and a batch is a rectangle, so short rows get filler tokens until every row matches the longest. That filler is called padding, and this function builds three parallel lists out of it.

• ii is the token ids, padded with the tokenizer’s dedicated padding token.
• am is the attention mask: a 1 for every real token and a 0 for every pad. This is a third mask, the one that stops real tokens attending to filler.
• lb is the labels, padded with -100, so the filler is never graded.

Then torch.tensor turns each nested Python list into a tensor, which is a grid of numbers with a fixed shape. The DataLoader line hands out groups of four rows, reshuffles between epochs, and calls this function on every group.

Block 4: the optimizer and the schedule

opt = torch.optim.AdamW(model.parameters(), lr=2e-5, weight_decay=0.01)
total = (len(loader) // accum) * epochs
sched = get_cosine_schedule_with_warmup(opt, int(0.03 * total), total)

The call to model.parameters() hands the optimizer every trainable number in the model. That one argument is what makes this a full fine-tune rather than a LoRA run. The weight decay is a small pull on every weight toward zero at each step.

The middle line is easy to get wrong. The schedule counts optimizer steps, not batches, and only every fourth batch takes a step. So the total is batches divided by the accumulation count, times the epochs. Give the scheduler the wrong total and the cosine curve finishes in the wrong place, either bottoming out early or never reaching the bottom. The int(0.03 * total) is the 3 percent warmup from the table.

Block 5: the loop itself

with torch.autocast("cuda", dtype=torch.bfloat16):
    out = model(input_ids=ii, attention_mask=am, labels=lb)   # 1 FORWARD, 2 LOSS
(out.loss / accum).backward()                                 # 3 BACKWARD

The with line opens a block inside which the heavy arithmetic runs in bf16 while the stored weights stay 32-bit. That is mixed precision, switched on for exactly the region that benefits.

Passing labels into the model call makes the model compute the loss itself. It shifts the labels by one, runs cross-entropy at every position, skips every position marked -100, and returns an object whose .loss attribute is a single number. Moves one and two, in one call.

Then .backward() is move three. It works out a gradient for every parameter and adds it into that parameter’s own gradient slot. Note the division in front of it: four quarter-sized contributions summing to one whole.

if (step + 1) % accum == 0:
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)
    opt.step(); sched.step(); opt.zero_grad()                 # 4 STEP, 5 ZERO

Everything in this block happens once every four micro-batches. First the clipping, which is the formula from the gradient-norm section, with max_norm set to 1.0. The limit needs no tuning.

Then the three calls that end a step. opt.step() applies the update, move four. sched.step() advances the learning rate one position along the warmup and cosine curve. opt.zero_grad() is move five, wiping the slots so the next four micro-batches start from nothing.

The last two lines log and save. out.loss.item() pulls that number off the card into ordinary Python. save_pretrained writes the weights and config into a folder, and the tokenizer is saved beside them because a model without its tokenizer is unusable.

Get the same run from a trainer (Version B), with every config line explained

Nobody writes that loop in production. A trainer already contains it, plus evaluation, checkpointing, logging, resuming after a crash and running across several cards. What Version A buys you is the ability to read Version B with none of it a black box.

import torch
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer

mid = "Qwen/Qwen2.5-1.5B"
tok = AutoTokenizer.from_pretrained(mid)
# borrow the Instruct checkpoint's chat template so assistant_only_loss below can
# find the markers it needs. Same tokenizer, no new tokens, no resize.
tok.chat_template = AutoTokenizer.from_pretrained(mid + "-Instruct").chat_template
model = AutoModelForCausalLM.from_pretrained(mid, dtype=torch.float32)  # explicit fp32 again
ds = load_dataset("HuggingFaceH4/no_robots")     # this one has train and test splits

cfg = SFTConfig(
    output_dir="qwen15b-sft",
    # the mechanic
    assistant_only_loss=True,                    # mask everything but assistant turns
    eos_token="<|im_end|>",                    # what Qwen2.5's template closes turns with
    max_length=1024,
    # the knobs
    learning_rate=2e-5,                          # FULL fine-tune regime
    lr_scheduler_type="cosine", warmup_steps=0.03,  # a float below 1 is read as a ratio
    num_train_epochs=1,
    per_device_train_batch_size=4, gradient_accumulation_steps=4,
    bf16=True, gradient_checkpointing=True,
    max_grad_norm=1.0, weight_decay=0.01,
    # watch it
    logging_steps=10, eval_strategy="steps", eval_steps=100,
    save_strategy="steps", save_steps=100, seed=42,
    load_best_model_at_end=True, metric_for_best_model="eval_loss", greater_is_better=False,
)

trainer = SFTTrainer(model=model, args=cfg,
                     train_dataset=ds["train"], eval_dataset=ds["test"],
                     processing_class=tok)       # NOT tokenizer=, which is the old API
trainer.train()
trainer.save_model("qwen15b-sft/final")

Every line in that config stands for something you have watched run.

The model is loaded with an explicit 32-bit dtype here too, for the reason given above, and it is the bf16=True config line rather than the loading call that switches on mixed precision. Drop the explicit dtype and this becomes a pure 16-bit run with no master copy, the same trap as before.

The chat template patch, in four steps

One line looks like an odd hack, and it is worth understanding rather than copying.

1. What the flag needs. Setting assistant_only_loss=True asks the library to grade only the assistant’s tokens. For that it needs the chat template to carry markers, written {% generation %}, wrapping the assistant’s part so the tokenizer can report which positions those are.
2. What happens without them. The trainer will patch in a known-good template, but it recognises templates by comparing the whole template string against ones it already knows. It does not read the template’s logic.
3. Why the base checkpoint misses. The base Qwen2.5-1.5B template differs from the Instruct one in exactly one string, its default system prompt. That is enough for the comparison to fail, so the trainer raises an error even though the base template is perfectly valid.
4. The fix. Copy the Instruct checkpoint’s template string onto the base tokenizer, the line in the script above. The two checkpoints share a tokenizer and a vocabulary, so this introduces no new tokens and needs no resize. It is a string swap.

Two fallbacks if you would rather not. Drop to Version A, which masks explicitly and always works. Or point chat_template_path at a training-ready template from another model, such as chat_template_path="HuggingFaceTB/SmolLM3-3B", which the library’s own documentation suggests, bearing in mind that this one does add new special tokens and resize the embedding matrix before you have trained a step.

Two end tokens, and why you name both

The eos_token line is the other one that looks like decoration and is not.

A model stops talking when it produces its end token. Qwen2.5 has a complication: its tokenizer’s default end token is <|endoftext|>, while its chat template closes every assistant turn with a different token, <|im_end|>. Those are two separate vocabulary entries.

So the training data teaches the model to finish an answer by emitting the second, and the default configuration is watching for the first. Unless you say otherwise, generation never sees the stop signal the model was trained to produce, and the model keeps writing. Naming the template’s own token in the config keeps training and stopping consistent.

What drifts in this API, and what to pin

This surface changes between releases, and four details above are the current spellings rather than the ones in older tutorials.

• Data arguments such as the maximum length live on the config object, not on the trainer.
• The tokenizer is passed as processing_class=. The older tokenizer= is not the current name.
• The length knob is max_length=, not max_seq_length=.
• There is no warmup_ratio argument any more. Use warmup_steps, where a float below 1 is read as a ratio of the total steps.

Pin your library versions and check the current documentation before a real run. The names above are current as of August 2026.

Watch three numbers while the run is going

Three things are worth a glance, and only one of them is the loss.

Training loss. Watch the shape more than any value. A smooth descent that flattens out is healthy. A jagged or rising curve means the learning rate is too high, and a nearly flat one means it is too low. For this model and dataset a run typically starts around 1.5 to 2.5 and flattens approaching 1.0, illustrative for this setup rather than a target. A loss heading for zero is not a triumph, and the cheating-model section says why.

Gradient norm. The single scalar derived earlier, logged every step and read against the clip limit of 1.0. It moves before the loss curve does. Comfortably under the limit with clipping on the odd batch is healthy: an unusual example is being caught, which is the point. Clipping on nearly every step means the raw gradient is consistently larger than the model can absorb, so lower the learning rate rather than raising the limit, which only hides it. A norm near zero from the first step means the steps are too small to move the model off the base. The full symptom table is the diagnostic matrix further down.

GPU memory. Near full on the training card is expected. What is a warning sign is a second card starting to move when you did not intend to use it, which means your device isolation did not take effect.

On timing: for 10,000 examples at an effective batch of 16 and a sequence length of 1024, one epoch of full fine-tuning on a single modern card takes on the order of tens of minutes. Your first run’s job is not a great model. It is a correct run you understand end to end.

Prove the model actually changed

Here is the trap that separates people who have run a fine-tune from people who understand one. The training loss went down, so it worked. Not necessarily. A falling training loss proves one thing: the model fitted the signal on the tokens you graded. Whether that produced the behaviour you wanted is a separate question, with two answers.

The first is a second loss, measured on data the model never trained on. That set is the holdout, and Version B computes it every 100 steps.

While both curves fall, the model is generalising, meaning it is learning the pattern rather than the examples. When the training loss keeps falling and the held-out loss turns upward, it has started memorising, and further epochs make it worse on anything new. On a 10,000-example set that turn can come quickly, which is why the recommendation starts at one epoch.

So your model is the checkpoint at the held-out minimum, not the one from the last step. Version B’s load_best_model_at_end line does that bookkeeping for you.

The before-and-after check

Loss curves are necessary and not sufficient. The real proof is generating from the base model and from yours, on prompts neither trained on, and reading the answers side by side.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

tok = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-1.5B")
msgs = [{"role": "user", "content": "Give me three names for a coffee shop near a data centre."}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True)  # dict form carries the mask
prompt_len = enc["input_ids"].shape[1]

eos_id = tok.convert_tokens_to_ids("<|im_end|>")     # Qwen2.5's real end-of-turn token

for path in ["Qwen/Qwen2.5-1.5B", "qwen15b-sft/final"]:      # base first, then yours
    m = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).to("cuda")
    out = m.generate(**enc.to("cuda"), max_new_tokens=120, do_sample=False,
                     eos_token_id=[tok.eos_token_id, eos_id])  # the default AND the template's
    print(f"\n=== {path} ===\n", tok.decode(out[0][prompt_len:], skip_special_tokens=True))

Three details are worth naming. This call asks for return_dict=True, the opposite of the training script, because here the attention mask that comes with the dictionary is wanted. The do_sample=False makes generation deterministic, so the comparison is between two models rather than two dice rolls. And eos_token_id names both stop tokens from the subsection above.

The shape of the contrast, illustrative rather than a captured run, looks like this.

=== Qwen/Qwen2.5-1.5B (base) ===
Coffee shop names can come from many places: your neighbourhood, a favourite
memory, or a pun on coffee itself. Think about the feeling you want customers
to have, then brainstorm broadly before narrowing down to a final choice.

=== qwen15b-sft/final (fine-tuned) ===
1. Uptime Coffee
2. Redundant Grounds
3. The Failover Cafe

Read that against the twelve layers. Nothing in the second answer needed a fact the base model lacked. What changed is layer 7, the shape of the answer, and layer 9, taking the request as a request rather than as a topic.

If your fine-tune answers correctly and then keeps going, check the end token setup first: it is the more common cause and the quicker to rule out. Only then check whether the closing token was inside the graded region during training, because a model that got no gradient signal on it has no reason to stop.

The failure the loss cannot show you

Full fine-tuning moves every weight, so teaching it your format can erode abilities it already had: arithmetic, code, general knowledge, which is to say layers 4 and 5. This is catastrophic forgetting, and your held-out loss will not reveal it, because that holdout is made of text that looks like your training data.

Catching it needs a separate probe on tasks that look nothing like your data, even a handful you check by hand. You do not have to fix it here. You need to see that it exists, because it is the problem LoRA reduces by freezing the base model. Part 15 turns that probe into a procedure: score the same general tasks on the base and on your candidate, and read the difference rather than the bare number.

The other half of the answer is upstream. Every choice here assumed the data was good, and that assumption does more work than any hyperparameter in the table. Part 10 is about the data itself.

Run the whole project, from Stage 0 to the decision

Everything above is one run. A project is a sequence, and the order is most of what keeps it honest. The stages are numbered from zero, because the first happens before you have anything to train.

Stage 6 is the one people skip and the one that saves days. If a deliberate overfit on 50 examples cannot drive the loss near zero, the problem is the loop or the mask, and no amount of real training will fix it.

Stage 8 has three parts because one measurement cannot see all three ways a fine-tune fails.

1. In-distribution. Held-out loss and a task score on data that looks like your training data. Says whether it learned the job.
2. Out-of-distribution. A probe on tasks that look nothing like your data. Says whether it forgot something, and nothing else in the pipeline can see that.
3. Behavioural. Real generations from the base and the candidate side by side, on prompts neither trained on. Says whether a person would call it better.

None of the three substitutes for the others. Stage 9 compares all three against the baseline written at Stage 0. If you cannot say whether to ship, the missing piece is almost always Stage 0 rather than the model.

A diagnostic matrix for a run that has gone wrong

Read the loss, the gradient norm and one sample of output together. Individually they are ambiguous. Together they identify most first-run failures.

Two habits make most of this table unnecessary. Change one setting at a time, and read the gradient norm alongside the loss rather than after it.