Masking in LLM Training: Loss, Causal and Padding

Read any mask as one keep-or-ignore flag per position

Strip away the variants and a mask is a list of flags, one per position, riding alongside your data. Each flag says keep or ignore. Some operation reads that list and honours it. That is the whole idea.

A position is just a slot in the sequence, counted from the left. Text goes into a model as tokens, which are chunks of text of roughly word size, and each token occupies one position. Six tokens means six positions, so a mask over them is six flags. Here is one:

The first three tokens are a question somebody asked. The last three are the answer. The flags say that only the answer counts. Nothing about that table is specific to machine learning, and nothing about it is hard. Every mask in this part is this picture pointed at a different operation.

So when you meet the word, two questions decode it. Which operation does this mask gate? And what does hiding a position buy you there? Answer both and the confusion goes away, because the three maskings give three different answers.

• Loss masking gates the scoring. Hidden positions are not graded, so the model learns nothing from them. You set this one.
• Causal masking gates attention. Each position is stopped from looking at anything later in the sequence. The model does this on its own, always, and you never touch it.
• Padding masking gates both. Filler tokens added to square off a batch are hidden from attention and from the loss. You set it, though the tools mostly set it for you.

Say what a label is, and why -100 removes a position from training

Start with the mask you configure directly, because it is the one that decides what your fine-tune actually learns.

A label is the right answer at one position

Training compares what the model predicted against what was correct. The correct answer at a position is called its label. For a language model the correct answer is always the same thing: the token that actually came next. So you never invent labels for a language model. The labels are a copy of the tokens themselves.

The copy is lined up so that the guess made at one position is scored against the token at the next one. Hugging Face’s models perform that shift for you inside the model, which is why the array of labels you hand over is exactly the same length as the array of tokens, holding exactly the same values. That sounds pointless until you realise what it buys: a second array you are free to edit.

Editing it is loss masking. Take the copy and overwrite every position you do not want graded with the value -100.

Here is the running example of this part: the same six words as above, now with the turn markers a real chat format inserts around them. It is one short instruction-and-response pair of the kind that fills HuggingFaceH4/no_robots, the roughly 10,000 human-written pairs this series fine-tunes on, tokenized the way Qwen2.5-1.5B expects, a model whose family Qwen’s own technical report describes in full. Other models use different markers for the same job. The system turn and the whitespace tokens are dropped here so the table fits on a screen.

Three of those columns become three arrays that travel together into the model, and they are the three names you will see in code. The tokens are stored as integer IDs in an array called input_ids. The label column is an array called labels. The last column is the attention_mask, which the section on padding below is about. Every position in this example holds a 1 there, because nothing has been padded yet.

Read the label column top to bottom and you have the mask. Eight positions marked ignore, four graded. Position 8 is worth a glance: the model is not graded on producing the word assistant, because at inference time, meaning whenever the model is actually answering somebody, that marker is handed to it rather than generated. Position 12 is graded, and the next section explains why that single row matters more than it looks.

You rarely write labels one position at a time. A collator does it. A collator is the small piece of code sitting between your dataset and the model: it takes several examples, pads them to a common length, stacks them into one batch, and builds the label array and the attention mask that travel with them. Libraries ship collators, and you can write your own. The ones people write by hand are where masking bugs are born.

What “ignore” actually does to a weight

Being told a position is ignored is not the same as knowing what happens. Follow the chain in four steps. It is short, and it is the entire point of the -100 convention.

Step 1. The loss is a sum over graded positions. Part 1 built cross-entropy from scratch: at each position you graded, look up the probability the model gave to the correct token and take the negative logarithm of it. High probability gives a small penalty, low probability gives a large one. Add those penalties over the graded positions and divide by how many there were. One number for the example.

Step 2. -100 is a value no real label can hold. Token IDs are positions in the model’s vocabulary list, so they run from zero upwards and are never negative. That makes any negative number safe to use as a marker. PyTorch’s cross-entropy has a setting called ignore_index, which names one label value meaning skip this position entirely, and its default value is -100. So a position holding -100 is dropped from the sum in step 1, and also from the count you divide by.

Step 3. Dropped from the sum means it cannot move the total. This is the step people skip. A gradient answers one question about one weight: if I nudge this weight, does the loss go up or down, and how fast? If a position never enters the sum, then nothing the model predicted at that position can change the total, however wrong it was. Its contribution to every weight’s gradient is exactly zero.

Step 4. Zero gradient means zero weight change. Weights only ever move through the optimizer, and the optimizer only ever acts on gradients, as the previous part on AdamW walks through line by line. So a masked position contributes no loss, therefore no gradient, therefore no update. The model is not being gently discouraged from learning that position. It is never taught by it at all.

One honest caveat on that chain. A masked token is still read. It still runs through the forward pass, and it still acts as context that later positions attend to, which is exactly what you want for a prompt. Masking removes a position from the scoring, and it removes nothing from the input. That also means masking is free: the compute is identical either way, because those tokens were going through the model regardless.

Choose which positions your run should grade

The default for instruction tuning is to grade the assistant’s tokens and nothing else. That convention comes from the papers that established instruction tuning: Wei et al. in FLAN and Sanh et al. in T0 both rewrote large collections of tasks as an instruction plus a response, and in both the response is the part the model is scored on. It is worth seeing why, because there is a real case where the opposite is correct.

You could grade every position. For raw pretraining, and for continued pretraining on a pile of plain domain text, that is exactly what you do, because there is no prompt and no response, just text the model should learn to continue. That is the objective Brown et al. trained GPT-3 with: every token in the corpus is graded, because every token is one the model should learn to predict. Instruction tuning is a different shape. An example is a conversation, and only one participant in it is you.

Three concrete costs come with grading the prompt.

It spends capacity on text you do not control. Every graded position teaches the model to produce that text. Prompts come from your users, in their phrasing, and you never want the model generating them.

It drowns the signal you care about. Prompts are usually much longer than responses, and much harder to predict. Take a plausible example: a 200-token prompt on which the model averages a penalty of 2.0, and a 40-token response on which it averages 1.0. Grading everything gives a total of 200 times 2.0 plus 40 times 1.0, which is 440, spread over 240 positions. The response accounts for 40 of those 440 points, about 9 percent. Roughly nine tenths of the pull on the weights that step is going into predicting the user’s words. Mask the prompt and it is 100 percent.

It can teach the wrong reflex. A model graded on continuing prompts learns to continue prompts. With a diverse instruction set that shows up as a model which extends your question instead of answering it.

Grade the closing token, or the model never learns to stop

An end-of-turn token is a special token the chat format puts at the end of each turn, marking where an answer finishes. In the table above it is the marker at position 12. It is a token like any other, so the model has to learn to produce it, and the only way it learns is by being graded on it.

Mask that closing token and you get a symptom that looks exactly like a quality problem. The fine-tuned model answers correctly, then keeps going: another sentence, an invented follow-up question, a second answer, on until it hits whatever length limit the caller set. Nobody suspects the collator, because the loss curve was fine and the answers are good. The graded region has to include the closing token. Make it the first thing you check when a tuned model rambles.

Grade every assistant turn, and keep the mask honest under packing

A single-turn example has one boundary and one graded span. Find where the answer starts and you are done. A multi-turn conversation has several, and the tempting shortcut is to grade only the final assistant turn, because the end of the sequence is the easiest place to find.

That shortcut throws away supervision you already paid for. In a conversation that runs user, assistant, user, assistant, with replies of 40 and 60 tokens, grading only the last one grades 60 tokens and discards 40. The first reply is just as valid a lesson. Grading all of them also teaches the model to hold its role across turns instead of drifting into narrating both sides of the conversation.

The mechanism does not change. Mask every system turn and every user turn, keep every assistant turn, and keep each turn’s closing token. If you build labels by hand, walk through the turns in a loop. Do not compute a single boundary.

Where the boundary comes from

Two pieces of tooling decide how hard this is, and both are worth naming.

A tokenizer is the component that splits text into tokens and maps them to integer IDs, and back again. Every model ships its own. A chat template is a rule that ships alongside it, turning a list of role-and-content messages into the exact text and special tokens that model expects, turn markers included. The template is where the boundary between prompt and response physically lives, so anything that finds the boundary is really reading the template.

Some templates go further and emit generation markers. These are small pieces of bookkeeping wrapped around each assistant turn, so that a tokenizer asked for them hands back an extra row of flags saying precisely which positions are assistant tokens. That row is the loss mask, worked out by the template author instead of by you.

Modern trainers use this. The flag completion_only_loss covers datasets stored as a prompt and a completion. The flag assistant_only_loss covers conversational datasets and finds every assistant span rather than the last one, which is why it needs those generation markers to exist in the first place. Whether your base model’s template has them is a real selection criterion, and one that the next part, on choosing a base model and building a bench, covers alongside the rest. Pin your library versions before you rely on any of these flag names, because this corner of the API has moved more than once.

The one setting that quietly invalidates your mask

Sequence packing concatenates several short examples end to end into one full-length sequence, so the card stops spending compute on padding. It is a genuine speed win and it changes the geometry every mask in this article assumes.

Two things have to be reset at each example boundary inside a packed sequence. The first is attention: without a reset, tokens from the second example can attend back into the first, because the causal rule only says a position may look leftwards, and after packing there is more to the left than there used to be. The second is position indices, the numbers telling the model where each token sits counted from the start of the sequence. Left alone, the second example is told it begins at position 900 rather than position 1.

Libraries that support packing handle both, and the ones that do not will happily train nonsense with no error. The part on data covers packing properly. The rule to carry away is narrow and useful: after you turn packing on, verify the mask again from scratch.

Verify the mask on one batch before you trust a run

This is the single cheapest habit in the pipeline, and it takes about thirty seconds. Build one batch, then look at it. Decode the graded positions back into text and confirm they are exactly the assistant’s words, nothing more and nothing less.

# sanity-check ONE example's mask. Do this every time you change data code.
labels = batch["labels"][0]
graded = labels[labels != -100]            # the tokens that actually count

print(tokenizer.decode(graded))            # should be ONLY the assistant reply
assert (labels != -100).any(), "mask is all -100, this batch teaches nothing"

# and the reverse check: what did we throw away?
masked_positions = (labels == -100).sum().item()
print(f"{masked_positions} of {len(labels)} positions masked")

If you have never used PyTorch, here is every line.

1. batch is what the collator produced: a dictionary whose keys are names such as input_ids, attention_mask and labels. Each value is a tensor, which is simply a grid of numbers. This one has a row per example in the batch and a column per position.
2. batch["labels"][0] pulls out the labels tensor and then takes row zero, the first example. Counting starts at zero, so row zero is the first row.
3. labels != -100 does not return a single true or false. It compares every position at once and returns a tensor of the same shape holding true or false per position. That is the mask, made visible.
4. Putting that true-or-false tensor inside the square brackets of labels keeps only the positions where it is true. So graded is the list of label IDs that will actually be scored.
5. tokenizer.decode turns a list of token IDs back into readable text. This is the line that matters. Read the output. It should be the assistant’s reply, and it should end with the closing marker.
6. assert stops the program with your message if the condition is false. .any() is true when at least one position is true, so this fires when every single label is -100.
7. (labels == -100).sum() counts the ignored positions, because summing true-or-false values counts the trues. .item() takes that one-element tensor and hands back a plain Python number you can print.

Two silent failure modes make the assertion worth keeping forever.

The first is truncation. Truncation is cutting an example off at the maximum length you configured. If a prompt is long enough that the cut lands before the response begins, that example’s labels are all -100 and it teaches nothing at all. No error is raised, because an example with nothing to learn from is not an error to any library.

The second is an off-by-a-few boundary. Shift the split by three tokens and you quietly grade the tail of the prompt along with the answer. The loss curve is indistinguishable.

The assertion above catches the first mode only when a whole batch is empty. That is not enough on its own. Filter examples whose labels are entirely -100 out of the dataset before training starts, so a handful of them cannot hide inside otherwise healthy batches. The part on running a full supervised fine-tune shows the boundary calculation that gets the split right in token space rather than in characters.

Explain the causal mask, and why negative infinity gives exactly zero attention

The second masking is built into the model. To see what it does, you need one paragraph on attention.

Attention is the step where each position looks at the other positions and mixes in whatever it finds useful. It works in three moves. First, for every pair of positions, the model computes a raw score saying how relevant one is to the other. Second, it runs softmax across each row of those scores, which turns them into weights that are all positive and add up to 1. Third, each position takes a blend of the other positions’ contents using those weights. The part on multi-head attention covers the machinery in full.

Now the problem the causal mask solves. During training the whole sequence goes through the model in one pass, for speed, and every position is simultaneously being trained to predict the token after it. Feeding the model the true text rather than its own guesses is called teacher forcing, and Lamb et al. named its cost in Professor Forcing: a model trained only on true prefixes has never had its own output as input. If position 5 were allowed to attend to position 6, it would be reading the very answer it is being graded on. It would score beautifully and learn nothing.

So the rule is imposed: a position may attend only to positions at or before its own. Draw the grid with queries down the side and keys across the top and the allowed region is the lower triangle.

Why the blocked cells are set to negative infinity

The blocked cells are not deleted. They are set to negative infinity, and then the softmax runs as usual. That sounds like a strange way to say no. Work it through and it is the neatest possible way.

Softmax, built from raw scores in Part 1, does two things in order. It raises e, about 2.718, to the power of each score, which makes every value positive. Then it divides each result by the total, which makes them add up to 1. Take three positions with scores 2.0, 1.0 and 0.1 and no mask at all:

• Raise e to each score: 7.39, 2.72, 1.11.
• Add them: 11.22.
• Divide each by the total: 0.66, 0.24, 0.10.

Now block the third position by replacing its score with negative infinity, and redo the same arithmetic. Raising e to a large negative power collapses fast: e to the power of minus 1 is 0.37, e to the power of minus 10 is 0.000045, e to the power of minus 100 is a decimal point followed by forty-three zeros. Push it all the way and the answer is exactly 0.

• Raise e to each score: 7.39, 2.72, 0.
• Add them: 10.11.
• Divide each by the total: 0.73, 0.27, 0.00.

The blocked position gets a weight of exactly zero, so none of its content reaches the output. The two survivors still add to 1, and they have shared out the blocked position’s weight between them in proportion to their own scores. Nothing downstream needs a special case, and nothing needs to know a mask was applied.

That is why negative infinity and not, say, zero. A score of zero would give e to the power of zero, which is 1, a perfectly respectable weight. Negative infinity is the one value softmax turns into no contribution at all, which lets the mask be applied by plain addition before the softmax and then forgotten. In real code libraries add the largest negative number the format can hold rather than true infinity, because arithmetic on actual infinities can produce a not-a-number value that poisons the run. The effect is the same to more decimal places than anything cares about.

The practical takeaway is short. You never set this mask. It is part of the architecture of any causal language model, applied inside every attention layer, on every pass, whether you are training or serving. The masks you own are the ones that depend on your data: where the prompt ends, and how long each sequence is.

Pad a batch without changing what it teaches

A batch is several examples processed together, and a GPU wants that batch as a rectangle: the same number of columns in every row. Real examples are different lengths. So you pad, which means appending a filler token to the short ones until every row reaches the length of the longest.

Padding is bookkeeping for the hardware. It carries no meaning, so you have to tell the model to disregard it, and the thing that does the telling is the attention mask: a 1 for each real token and a 0 for each pad.

The zeros work through the same machinery as the causal mask. A column marked zero is turned into negative infinity before the softmax, so every real token gives it a weight of exactly zero. Without that, tokens would blend meaningless filler into their own representations, and the amount of nonsense mixed in would depend on how long the longest unrelated example in the batch happened to be.

Padding also overlaps with loss masking, and this is the part people forget. A pad position needs two markings, not one: an attention mask of 0 so nothing attends to it, and a label of -100 so it is not graded. In a collator that is two lines, extend the attention mask with zeros and extend the labels with -100. Miss the second and your model spends part of every step being graded on its ability to predict filler.

Which side you pad on, and why it costs people a day

Padding has a side, and the right side differs between training and generation. This is the highest-value paragraph in the article for anyone who is about to generate text from their own fine-tune.

Generation works by appending. You hand the model a prompt, it produces one token, that token is written into the next column of the row, and the loop runs again from there. The model always continues from the end of the row. Now put a four-token prompt into a six-column batch two ways:

Read the last column. With right padding, the sequence the model is asked to continue reads prompt, filler, filler, and then its own new tokens land on the far side of that filler. The attention mask does stop the pad tokens being attended to, and it still does not help, because the model is being asked to write the continuation in the wrong place. With left padding every row’s real final token sits flush against the right-hand edge, which is exactly the position generation continues from, and every row in the batch lines up.

So: left-pad for generation, right-pad for training. Training has no appending loop, every position is scored where it sits, and right padding is the ordinary convention. Generation libraries will usually warn you about right padding on a decoder-only model, which is a model that only ever looks leftwards. Heed the warning. A wrong padding side produces garbage output from a perfectly good model, and there is nothing in the model or the training log to point at.

See all three masks fire in one forward pass

In a single training step all three run, and they never interfere, because they gate different operations at different moments.

Inside every attention layer, the causal mask and the padding mask are combined into one grid of allowed and blocked cells, and applied together in the same addition before the softmax. A cell survives only if both agree: the key position is at or before the query, and the key position is a real token. Then, at the very end of the forward pass, once the model has produced a guess at every position, the loss mask decides which of those guesses are graded at all.

They compose without conflict because they answer different questions. Causal and padding answer which positions may this token look at. Loss answers which positions do we score. Here is the whole thing in one table.

Optional: recognise a fourth meaning of masking in other models

This section does not apply to fine-tuning a causal language model. Skip it if you like. It earns its place only because the word turns up in one more sense and the confusion is common.

Masked language modelling is the training objective behind BERT-style models. About 15 percent of the input tokens are replaced with a placeholder, and the model is trained to reconstruct them using context from both sides. There is no causal mask involved, because attention is deliberately allowed to look in both directions. Nothing is being hidden from the loss. Something is being hidden from the input, which is the opposite arrangement to everything above.

It is a pretraining objective for understanding models, the kind used for classification, search and embeddings, rather than for generating text. You will not use it to fine-tune a chat model. It does matter when you build a retrieval pipeline with an embedding model, because that embedding model was very likely trained this way and your decoder model was not. When someone says masking and means “predict the hidden word”, this is the sense they mean.