Data Is the Actual Job: Formats, Quality, Splits and Synthetic Data

Pick the file format that grades the tokens you care about

Before quality comes shape. Start with one example, written out in plain English.

The instruction is “Rewrite this in plain English: The applicant is required to submit the form.” The answer is “You need to send us the form.”

That single example can be stored three different ways. The three are not interchangeable, because they hand the training loop three different jobs. One word has to come first.

What “graded” means

Training walks through an example one token at a time. A token is a chunk of text, usually a word or a piece of one. At each position the model guesses what comes next. That guess is scored against the token that really came next. The score is called the loss, and lower means the guess was closer.

Not every position has to be scored. You choose. A position that counts toward the loss is graded. A position that does not count is masked out, and the model learns nothing from it. Deciding which positions count is called loss masking, and Part 7 in the prerequisites above is entirely about it.

Hold on to that, because your file format picks the masking for you.

The three formats

Raw text. One field, usually called text, holding the instruction and the answer glued together as one string. Every token is graded, including the instruction.

Prompt and completion. Two fields, prompt and completion. The instruction goes in one, the answer in the other. Trainers grade the completion and skip the prompt.

Conversational. A list of messages, each with a role and some content. The roles are system, user and assistant. A system message sets the rules, a user message asks, an assistant message answers. You grade the assistant messages and skip the rest.

Now the consequence. Raw text grades the instruction as well as the answer. So the model is being taught to write instructions too. Ask it a question later and it will answer, then cheerfully write the next user question by itself, because that is a thing it saw and was rewarded for. Use raw text only when you want the model to soak up a body of domain writing. Never use it for instruction tuning.

Instruction tuning means training a model on instruction and answer pairs so it learns to follow instructions. That is what the rest of this part assumes you are doing.

Two details that break real runs

First, multi-turn conversations. If an example has three assistant replies, grade all three. Grading only the last one throws away two thirds of the teaching. It also lets the model drift into narrating both sides of a conversation, because it never saw the earlier replies rewarded.

Second, the stop marker. Every chat model has a special end-of-turn token that means “the answer stops here”. It has to sit inside the graded region. Leave it outside and the model never learns to stop talking. It will answer your question and then keep going, forever, which looks like a decoding bug and is not one.

The thing that lays those markers down is the chat template. It is a rule that ships with the model’s tokenizer, and it turns a list of messages into the exact text and special markers the model expects. Use the template that came with your model. Writing your own string format is the fastest way to teach a model a structure it will never see again at serving time.

Pack short examples without letting them leak into each other

Most instruction examples are short. Most training runs use a fixed maximum length. Those two facts cost you money, and the fix has a trap in it.

Say your maximum length is 1,024 tokens and your typical example is 200. Every sequence needs to be the same length for the hardware, so the other 824 positions get filled with padding: meaningless filler tokens that exist only to make the shapes line up. They are masked out of the loss, so they teach nothing. You are paying for 1,024 positions of compute and getting 200 positions of teaching. That is about 80 percent waste.

Packing fixes it by gluing several short examples end to end into one full-length sequence. Five 200-token examples fill the 1,024 almost exactly. Now nearly every position is real text.

Here is the trap. Inside the model, each token looks back at the tokens before it and pulls in whatever seems useful. That step is called attention, and it is what lets a model use context. Glue two examples together naively and the second one can look back into the first.

Picture it concretely. Example A is a recipe. Example B is a Python question. Packed naively, the tokens of the Python answer can see the recipe sitting behind them. The model quietly learns that Python answers sometimes follow recipes. Nothing errors. The loss looks fine. You have just taught the model a relationship that does not exist.

There is a second half to the trap. Each token also carries a number saying where it sits in the sequence. If example B starts at position 500, the model treats B’s first word as the 500th word of one long document. Correct packing resets that count at every boundary, so each example starts at position zero again.

In current TRL you get correct packing by setting packing=True in SFTConfig. The default packing_strategy="bfd" also switches on padding-free training, which flattens the batch into one continuous run of tokens. Per TRL’s own documentation that path needs a FlashAttention 2 or 3 backend, because that backend is what knows how to keep the flattened pieces apart. FlashAttention is a faster way of computing attention, covered in Part 3 of this series.

Two practical rules. Pack when you have many short examples and the padding waste is costing real time. Do not pack on your first learning runs, because it is one more thing that can corrupt the signal silently while you are still checking that your masks are right. And pin your library versions before you build anything on these flags, because packing and masking options have moved between releases more than once.

Judge a dataset by quality before you judge it by size

Here is a decision you will actually face. You have written 200 examples yourself, and they are good. A colleague offers 2,000 more scraped from a support forum. Maybe half of those are decent. Do you take them?

Instinct says yes, because more data is better. For supervised fine-tuning, usually it is not. The same instinct fails at pretraining scale too. Gunasekar and colleagues trained a 1.3B-parameter code model on a small curated corpus of textbook-quality text, and reported it beating much larger models trained on far more scraped code. Work out why from what training does.

The loss is averaged over the graded positions in the batch. So every example gets a vote on what the model becomes, and the votes are equal. Add 1,000 sloppy examples to 200 good ones and the sloppy ones now cast 83 percent of the votes. You have not added a little noise on top of your signal. You have made noise the majority position.

The reason this bites harder for fine-tuning than you might expect is that a fine-tune is not teaching facts. The model learned its facts during pretraining, over trillions of tokens. To picture the gap, Gao and colleagues assembled The Pile out of 22 separate sources into a corpus of roughly 825 GiB of text, and Brown and colleagues trained GPT-3 on a few hundred billion tokens. A few thousand examples cannot compete with that and are not trying to. What they install is a habit: the shape of the answer, its length, its tone, when to stop. Habits are learned from a few clean demonstrations and wrecked by many dirty ones.

The two published results everyone cites

Two studies made this concrete, and they are worth stating carefully rather than as folklore.

LIMA. Zhou and colleagues fine-tuned a 65B model on exactly 1,000 carefully written examples. Human raters then compared its answers against three other systems, one comparison at a time. LIMA’s answer was rated equal or better in 43 percent of comparisons against GPT-4, 58 percent against Bard, and 65 percent against DaVinci-003. Read those numbers as “how often the 1,000-example model held its own”. Against the strongest opponent in the set it still tied or won more than four times in ten.

AlpaGasus. Chen and colleagues started from a noisy 52,000-example instruction set. They asked a strong model to score every example, kept about 9,000, and threw the rest away. The model trained on the 9,000 was rated better than the model trained on all 52,000. It also trained about 5.7 times faster, cutting a 7B fine-tune from roughly 80 minutes to roughly 14.

The rule that falls out: adding data helps only while the new data is at least as clean as what you already have. Past that line, every extra example teaches the model your noise. So the honest answer to your colleague is “yes, if I can filter them first”, and the rest of this part is about how.

What “quality” decomposes into

Quality is not a feeling. It is five properties you can check, and each one has a failure it prevents.

Two of those names need unpacking. Mode collapse is when tuning crushes the variety out of a model, so it produces the same phrasings over and over. You only catch it by reading many outputs, never a single sample. Length bias is a systematic pull toward one answer length that has nothing to do with the task. It usually arrives from raters preferring longer answers, and you can also install it directly by making all your own answers the same size.

Run the three cleaning passes, and run them in the right order

Real pipelines spend most of their effort here. Three operations do nearly all the work. The order they run in matters as much as the operations themselves.

Pass one: deduplication

Duplicates are not harmless filler. Go back to the voting picture. If one answer appears 50 times in a 1,000-example set, it casts 5 percent of the votes rather than 0.1 percent. It gets 50 times the pull on the weights that a single example would. The model memorises that answer and over-weights its style.

Deduplication runs in two passes.

Exact duplicates are easy. Turn each example into a fingerprint with a hash function, which maps any text to a short fixed-length value, and identical text always produces the same value. Group by fingerprint, keep one from each group. This is fast and catches copy-paste.

Near duplicates are the real problem. Compare these two:

“write a python function that reverses a string”

“write a function in python that reverses a string”

Those are the same example. Their hashes are completely different, because one word moved. To catch this you need a similarity score rather than an equality test. The standard one is Jaccard similarity, and you can compute it by hand.

1. Take the set of words in the first text: write, a, python, function, that, reverses, string. That is 7 words.
2. Take the set of words in the second: write, a, function, in, python, that, reverses, string. That is 8.
3. Count the words in both: 7 of them.
4. Count the words in either: 8, the 7 shared plus “in”.
5. Divide. 7 divided by 8 is 0.875.

A score of 0.875 out of a possible 1.0 says these are near-copies. Set a threshold, say 0.8, and drop anything above it. Real tools chop the text into short runs of consecutive words rather than single words, which is stricter, but the arithmetic has exactly this shape.

One problem remains. Comparing every example against every other example means comparing every pair. For 10,000 examples that is about 50 million pairs, which is fine. For a million examples it is about 500 billion pairs, which is not. The standard escape is called min-hashing with locality-sensitive hashing. It builds a short fingerprint for each example whose chance of matching another fingerprint is roughly the Jaccard score you would have computed, then sorts examples into buckets so that only plausible matches ever get compared directly. You do not need to implement it. You need to know that a library such as datasketch exists for exactly this, and that near-duplicate removal without it does not scale.

The payoff is measured. Lee and colleagues found that more than 1 percent of the tokens a model produces unprompted are copied verbatim from its training set when that set was not deduplicated. Deduplicating cut the rate of memorised output by roughly a factor of ten.

Pass two: decontamination

Decontamination means checking that nothing in your training data also appears in the data you will measure on, and removing whatever does. Skip it and your reported number stops measuring skill and starts measuring memory.

Work the arithmetic once and you will never skip it again. Suppose you have 200 test questions. Your model genuinely answers 60 percent of unseen questions correctly. Now suppose some test questions accidentally got copied into the training set, so the model has simply memorised those answers.

• Nothing leaked. The score is 60 percent of 200, so 120 correct. Reported: 60 percent. Honest.
• 20 questions leaked, which is 10 percent. Those 20 are all correct. Of the remaining 180, 60 percent is 108. Total 128. Reported: 64 percent.
• 50 questions leaked, which is 25 percent. Those 50 are correct, plus 60 percent of the remaining 150, which is 90. Total 140. Reported: 70 percent.

The model never got better. In the last row you would report 70 percent for a model that is worth 60. That is why one leaked test item invalidates a benchmark: a benchmark is only a claim about unseen questions, and a leaked question is not unseen. A benchmark, here, means a fixed test set with a published score, the kind of thing people put on slides.

Decontamination in practice means scanning training text for overlap with your held-out set, and with any public benchmark you intend to quote. The usual check is n-gram overlap, which means looking for runs of the same consecutive words, typically eight or thirteen of them, appearing in both. Exact string matching is not enough, for the same reason exact deduplication is not enough. Reworded copies leak just as effectively.

This is not a rare hazard. The same deduplication study found train-test overlap affecting more than 4 percent of the validation set in standard benchmarks. Public benchmark questions also leak into pretraining data, which is why a public leaderboard score is a coarse filter and never a decision.

Pass three: quality scoring and filtering

You are not going to hand-read ten thousand examples. Score them instead, then keep a slice off the top. There is a ladder of rigour, and cheap comes first.

1. Heuristics. Length bounds, a language check, regular expressions for boilerplate such as “As an assistant” or leftover markup. Costs nothing and removes a surprising amount.
2. A scoring model. A reward model is a small model trained on human comparisons to give any answer a single number. Run it over the set and threshold.
3. A strong model as judge. Ask a capable model to rate each example against a written rubric. This is the technique AlpaGasus used to get from 52,000 to 9,000.

Judging with a model works better than it has any right to. Zheng and colleagues found a strong judge agrees with human raters more than 80 percent of the time, which is about as often as two humans agree with each other. It also has known habits: it favours the answer shown first, it favours longer answers, and it favours its own writing style. Randomise the order, and be suspicious when the winners are all the long ones.

Finally, protect diversity on the way out. Cluster the survivors and cap how many near-copies of any one cluster you keep. Otherwise a top-scoring pattern quietly takes over the set, and you have optimised your way into mode collapse.

Both orderings in that figure are load-bearing. Deduplicate first so you do not spend judge calls scoring fifty copies of the same example. Decontaminate before you split, because once the data is split, a leak has already crossed the wall and no later step will find it.

One more leak deserves its own name, because a duplicate check cannot see it. Label leakage is when information from the expected answer sneaks into the input field of an example. Imagine a support dataset where the ticket text quietly includes the resolution code, because the export ran after the ticket was closed. The model reads the answer off the prompt. Your metrics soar. In production the field is empty and the model is useless. Part 15 covers how to hunt for it.

Every one of these passes is also an auditable control. Being able to say you deduplicated, decontaminated against the evaluation set, and filtered with documented thresholds is exactly the evidence a model-risk review will ask for. Build it as a logged, repeatable script rather than a notebook you ran once.

Split the data three ways and keep the wall between the parts

You need three sets, not two. The reason is about how each one gets used up.

• Train. The model learns on it. Nothing measured here means anything, because the model has seen every answer.
• Validation. You check this during training to pick settings and to choose which saved copy of the model to keep. Keeping the copy from the point where validation loss was lowest is called early stopping.
• Test. You look at this once, at the very end, for one honest number.

Why validation cannot also be test is worth building slowly, because it is the part people talk themselves out of.

Say you try twelve learning rates and keep whichever scores best on validation. You have now made twelve decisions using that set. The winner is partly genuinely better and partly just lucky on those particular examples. Repeat that over a project and validation slowly turns into a set you have fitted to, one decision at a time. It gets used up. That is not a moral failing, it is arithmetic, and it is why the test set exists and stays sealed.

Tune against the test set even once and it becomes a second validation set. There is no ceremony that restores it.

Split by category, not at random

A random split can betray you, and here is the shape of it.

The word you need first is distribution, which just means the tally of how much of each kind of thing your data contains. The no_robots dataset is a good example. Its 10,000 examples carry a category label, and the tally is lopsided. Generation covers about 4,560 of them. Extract covers about 190.

Now take a random 5 percent test split, so 500 examples. On average about 9 or 10 of those should be Extract examples. On average. A random draw can easily hand you two, or none. If it hands you none, your test score says precisely nothing about extraction, and you will not notice, because the number will look fine.

A stratified split fixes this. Split within each category separately, then combine. Take 5 percent of Generation, 5 percent of Extract, 5 percent of every other category, and glue the pieces together. Every category then appears in every split in the same proportion it appears overall. Do the same by difficulty if you have difficulty labels, and by source if your data came from several places.

Keep one probe that looks nothing like your training data

Your validation and test sets are drawn from the same pile as your training set. So they can only tell you how the model does on data that looks like what you trained on. They are blind to everything else.

Everything else includes the thing most likely to hurt you. Fine-tuning hard on one task can quietly erode abilities the model already had, such as arithmetic or following a formatting instruction. That is called catastrophic forgetting. Your in-distribution test set will report success the whole time it is happening.

The fix is a small out-of-distribution probe: a handful of tasks deliberately unlike your training data, scored before and after the fine-tune. A few maths questions, a few general-knowledge questions, a couple of instruction-following checks. It costs almost nothing and it is the only instrument that sees this failure. Part 15 turns it into a full procedure.

The rule to carry away: any number you show someone is worth exactly as much as the wall between the set it was measured on and the set the model trained on. Write the split method down, version it, and never move the wall to make a number look better.

Generate synthetic data without collapsing the model

Human-written examples are slow and expensive. Synthetic data means training examples written by a model instead of a person. It is how many teams reach scale, and it is a genuine capability with a genuine failure mode attached.

Three reasons to generate at all. Scale, because a hundred thousand examples costs API calls rather than annotator hours. Targeted coverage, because if you need 500 examples of one rare edge case you can simply ask for exactly those. And distillation, where a strong model writes high-quality answers and a small model trains to copy them.

The failure that defines the topic: model collapse

Build this one up from marbles, because the mechanism is simpler than the name suggests.

Put 1,000 marbles in a bag. 990 are red and 10 are blue. Draw 100 at random. On average you expect 1 blue marble. Sometimes you get 2. Often you get 0.

Now throw the original bag away and refill it from what you drew, scaled back up to 1,000. If your 100 draws contained no blue, there is no blue in the new bag. Blue is gone. Not rare, gone. Repeat the whole procedure and the next-rarest colour goes the same way.

A model generating training data behaves like that draw. It produces common patterns often and rare ones rarely. The rare ones live in what statisticians call the tails of the distribution: the edge cases that are individually unlikely and collectively important. Train the next model on that output and the tails thin. Do it again and they vanish.

Shumailov and colleagues published exactly this experiment in Nature in 2024, and named the effect model collapse. They fine-tuned a small language model, OPT-125m, on the wikitext2 dataset. Then they used it to generate a dataset the same size, trained the next generation on that, and repeated for nine generations, five separate times.

Their headline measure was perplexity, which is one score for how surprised a model is by a piece of real text. Lower is better. Before fine-tuning, the model scored about 115 on the real test text. After fine-tuning on the real wikitext2 data it scored about 34, which is the model doing its job well. When each generation trained only on the previous generation’s output, later generations gave back 20 to 28 points of that gain.

The examples are more vivid than the numbers. Given a prompt about fourteenth-century English church architecture, the first model wrote a sensible paragraph about Perpendicular Revival architecture and St. John’s Cathedral in London. By generation nine, the same prompt produced text about the world’s largest populations of black-tailed jackrabbits, white-tailed jackrabbits, blue-tailed jackrabbits, red-tailed jackrabbits and yellow-tailed jackrabbits. The model has not become random. It has fallen into a groove and is repeating variations of one pattern.

The paper separates two stages. In early collapse the model starts losing the tails, so rare cases stop appearing. In late collapse the model converges on something with little resemblance to the original data, with much less variety left in it. Early collapse is the dangerous one, because a model in early collapse still looks fine on ordinary examples.

The primary cause they name is plain sampling. Any finite sample of a distribution loses some of the rare events, and information lost at one step can never come back at the next. That is the marble bag. Two secondary causes sit on top: the model cannot represent the original distribution perfectly, and the training procedure itself has its own biases.

One detail worth knowing, because it kills the obvious fix. The collapsed models repeat phrases constantly, so the natural response is to force variety with a repetition penalty. The researchers tried it. The models produced worse continuations, and perplexity doubled compared with the original. Suppressing the symptom made the underlying problem worse.

The defence is mixing, not abstinence

The same paper points at the fix. In a second setting they kept 10 percent of the original real data in every generation’s training set. The degradation dropped to minor. Real data anchors the tails that generated data drops.

So the working recipe for synthetic data is a pipeline, not a prompt.

1. Keep real data in the mix. Never train a generation purely on the previous generation’s output.
2. Deduplicate the generated set first. Models repeat themselves far more than people do, so near-duplicate rates in generated data are high.
3. Verify anything checkable. For maths, check the answer. For code, run the tests. A teacher model’s mistakes get copied faithfully and then amplified.
4. Then score and filter, and cap per cluster so one dominant pattern cannot swamp the set.

The useful way to hold it: generation sets the ceiling on possible quality, and curation decides how much of that ceiling you actually reach. Producing a million examples is a weekend of API calls. Making them usable is the job.

The licence follows the weights

One compliance point that is not a footnote. Data generated by a commercial model is governed by that model’s terms of service, and several of those explicitly forbid using the outputs to train a competing model. So “we distilled our instruction set from a frontier model” can carry legal exposure that travels with the resulting weights, long after anyone remembers where the data came from.

For a defensible pipeline, use permissively licensed teacher models or your own. Record the provenance of every synthetic batch: which model, which version, which prompt, which date. Keep that record with the dataset. It is precisely what a model-risk review asks for, and it is why a fully documented training-data story has real commercial value.

Poison a copy of your data on purpose and see what each fault breaks

Everything above is easier to believe once you have watched it happen. The protocol is small. Take a clean baseline dataset. Make several copies, and damage each copy in exactly one way, so only one variable moves at a time. Train on each. Compare all of them against the same clean held-out set and the same handful of generation checks.

The damage column below is reasoned from the mechanism rather than measured from one particular run. Treat it as what to expect, and confirm it against your own numbers.

The refusal row introduces one term. An alignment tax is capability lost in exchange for making a model better behaved. Usually it arrives from a deliberate safety decision. This row shows you can install it entirely by accident, through data hygiene alone.

Now the reason this protocol teaches more than any amount of reading. Every training curve in that experiment goes down. All of them. The training loss measures how well the model predicts the text it was given, and it has no opinion about whether that text was any good. Truncate every answer to one sentence and the model learns to produce one sentence, and gets steadily better at it, and the curve looks textbook.

Four of the five corruptions at least fail honestly: run a clean held-out evaluation and the damage shows up as a worse score. Contamination is the exception. It fails dishonestly, because the model memorised the leaked answers, so the reported number looks as good as baseline while the model is worse. That is why decontamination is the control you never skip, and why it has to run before the split.

Audit any dataset in one minute before you train on it

You do not need a purpose-built tool to catch most of this. Three questions cover the bulk of it. How many examples are actually here? How many are exact copies? And did anything in train leak into the evaluation split?

from collections import Counter
from datasets import load_dataset

ds = load_dataset("HuggingFaceH4/no_robots")
train, test = ds["train"], ds["test"]

def norm(example):
    joined = " ".join(m["content"] for m in example["messages"])
    return " ".join(joined.split()).lower()

train_texts = [norm(r) for r in train]
test_texts = [norm(r) for r in test]

print("train examples:  ", len(train_texts))
print("test examples:   ", len(test_texts))
print("exact duplicates:", len(train_texts) - len(set(train_texts)))
print("train/test leak: ", len(set(train_texts) & set(test_texts)))
print("categories:      ", Counter(train["category"]).most_common())

Line by line, assuming you have never used any of these libraries.

• from collections import Counter pulls in a standard Python helper that tallies how many times each value appears in a list.
• from datasets import load_dataset pulls in the Hugging Face datasets library, which downloads a named dataset and caches it locally.
• load_dataset("HuggingFaceH4/no_robots") fetches the dataset. It returns a dictionary-like object with one entry per split.
• ds["train"], ds["test"] takes the two splits out. Here that is 9,500 and 500 examples.
• def norm(example) defines a function that flattens one example into a single plain string, so that two examples can be compared.
• example["messages"] is the list of turns. Each turn is a small dictionary with a role and a content. The join glues every turn’s text together with spaces.
• " ".join(joined.split()) collapses every run of spaces, tabs and newlines into one space. .lower() then lowercases. Both exist so that two examples differing only in spacing or capitals are treated as the same example.
• The two list comprehensions run norm over every row and collect the results, giving one string per example.
• set(train_texts) builds a Python set, which silently discards repeats. So the list length minus the set length is the number of extra copies.
• set(train_texts) & set(test_texts) is set intersection: the strings that appear in both. Anything above zero is a leak, and it is the one line here you must never ignore.
• Counter(train["category"]).most_common() reads the whole category column as a list, tallies it, and sorts it by count. This is the lopsided distribution from the splitting section, printed out.

Extend the same shape in three directions when you need more. Add a near-duplicate rate by running min-hash over the normalised text and thresholding on Jaccard similarity, using a library rather than writing it. Add a format check by confirming every example survives the tokenizer’s chat template without an error. Add a length distribution by counting tokens per example with the exact tokenizer you are about to train with, because that is the only count that matches what training will see.

Read the numbers your own run prints rather than trusting any figure quoted here. They depend on the dataset, on how you chose to normalise the text, and on your similarity threshold.

Run it before every training job, not once. It costs a minute, and it catches the class of problem that otherwise appears as an unexplained quality regression three weeks later. This discipline pays off whatever you train next, including the low-rank adapters in the next part on LoRA.