Evaluating a Fine-Tune: The Ladder, the Delta and Six Silent Failures

Name the four questions a falling loss cannot answer

Picture a student revising for an exam. They work through a hundred practice questions, over and over, with the answers on the back of each card. By Friday they get all hundred right.

What have you learned about that student? Almost nothing. You do not know whether they can answer a question they have not seen. You do not know whether they still remember last year’s material. You do not know whether they are any use in a real job.

Training loss is that Friday score. It is measured on the very examples the model was trained on. Four separate questions matter, and loss answers only the first of them, weakly.

1. Did it learn the task at all? Did the behaviour you wanted actually go in?
2. Does it work on inputs it has never seen? Or did it only memorise your examples?
3. Did it break anything? Can it still do the things it could do before you touched it?
4. Is the output any good? Would a person reading it be satisfied?

Every one of the last three needs its own test. Each test has to run outside the training loop, on data the training never touched. That is the whole design principle of this part, and everything below is an application of it.

Test the model on data it has never seen

Go back to the flashcards. Your student scored a hundred percent on the hundred cards they studied. There are two completely different reasons that could happen.

• They understood the underlying pattern, so any similar question would also work.
• They memorised those hundred cards, front and back, and know nothing else.

Those two are the entire subject of this section. The second one is memorisation: storing the specific examples. The first is generalisation: performing well on inputs that were never in the training data. Generalisation is the thing you actually want. It is also the thing training loss structurally cannot measure, because training loss is only ever measured on the cards.

So you need cards the student has not seen. In machine learning that slice of data has a name: a holdout. A holdout is data you deliberately set aside before training and never show to the training loop. A score measured on data the model trained on is not evidence of anything. A score measured on a holdout is.

Three piles, not two

In practice you cut the data three ways, not two, and the third cut surprises people.

1. Training set. The model sees these and learns from them. This is the only pile that changes any weights.
2. Validation set. The model never learns from these, but you look at them during the run. This is where Part 9’s eval-loss curve comes from, and it is what you use to pick a learning rate or choose which saved checkpoint to keep.
3. Test set. You look at this once, at the very end, for one honest number.

Why three? Because looking at a set is itself a way of fitting to it. Suppose you try eight learning rates and keep whichever scored best on the validation set. You have now hand-fitted that set. It flattered you a little, and it will keep flattering you. The test set is the one pile you have not spent. Spend it once.

Make the split representative

A holdout is only useful if it looks like real traffic. Say you have 1,000 support tickets: 900 refund requests and 100 billing disputes. Take a random 10 percent as a test set and you might land 6 billing disputes, or you might land 16. Your entire billing score then rests on 6 examples, and 6 examples cannot tell you anything.

The fix is a stratified split: take 10 percent of each category separately. That gives 90 refunds and 10 billing disputes, every time, in the same proportions as the real pile. Part 10 covers the split mechanics in full.

Three more habits make a holdout worth having. Include the hard cases and the adversarial ones, meaning inputs deliberately written to trip the model, rather than only the easy examples that were pleasant to collect. Fix the set and version it, so a score from March is comparable to a score from June. And assemble it exactly as production does: same prompt template, same system prompt, same surrounding context. An evaluation that uses a different prompt format is measuring a model you are not going to ship.

Keep the evaluation set clean, and spot the two ways it leaks

Everything above collapses if the model has already seen the test items. Your student glanced at ten of the hundred exam questions last week. Their score goes up. Their knowledge does not.

That is contamination: overlap between the training data and the evaluation data, including reworded near-copies. It is the one data fault that fails dishonestly. A bad batch of training text makes your score fall, so you go and look. Contamination makes your score rise, so you celebrate and ship. Part 10 treats it as a data-pipeline control, which is where the fix belongs.

Public benchmarks have this problem permanently. If a test’s questions have ever been on the open internet, assume the pretraining crawl swallowed them years ago. The model may be reciting rather than reasoning, and the score cannot tell the two apart.

Detecting contamination after the fact is unreliable, and it gets worse once the training text differs in wording from the original question. Xu and colleagues survey the methods and the gaps. So do not lean on detection. Lean on a private set instead, built from your own data, never published, versioned, and treated as the only score that decides anything. Public benchmarks are a coarse first filter and never the deciding vote.

The quieter leak, inside a single example

There is a second kind of leak, and it hides inside one training example rather than across two files.

Say your ticket records carry a field called resolution_code. An agent fills it in when they close the ticket, so it exists in your archive and it always agrees with the correct answer. You build training examples straight from the archive and leave that field in the prompt. The model does not learn to read the ticket. It learns to read the field.

Your metrics soar. Then you deploy, and a real request arrives with that field empty, because nobody has closed the ticket yet. The model has no idea what to do. That is label leakage: information from the expected answer bleeding into the input side of an example.

The check is a single question, asked of every field in your prompt. Would this value exist at the moment a real request arrives? If not, delete it before you build a single example.

Turn a bare score into a delta you can defend

Your fine-tune scores 68 percent on a clean, private holdout. Is that good?

You cannot answer that. Nobody can. The number needs something to be compared against, and the right thing is the model you started from. So score the base model too, on the identical evaluation: same examples, same prompt template, same judge, same everything. Now you have two numbers, and the difference between them is the delta. Without a delta you cannot tell an improved model from a merely changed one.

Why a 2-point gain on 100 examples is probably nothing

Say your holdout has 100 examples. The base model gets 61 right. The fine-tune gets 63. A 2-point gain. Ship it?

No. Start with a coin. Flip a fair coin 100 times and you expect 50 heads. You will rarely get exactly 50. You get 47, or 54, and nobody claims the coin has changed. That wobble is what happens whenever you measure a small sample of something.

Your 100 examples are a sample too. You did not test on every ticket your users will ever send. You tested on 100 of them, drawn from a much larger pool. The honest question is this: if you had drawn a different 100, would the gap still be there?

Now look at the 2 points example by example, rather than as two totals. A realistic breakdown:

• 82 examples where both models scored the same, right or wrong together.
• 10 examples where the fine-tune was right and the base was wrong.
• 8 examples where the base was right and the fine-tune was wrong.

Ten minus eight is two, out of a hundred, which is the 2-point gain. But look at what it rests on. The 82 agreements carry no information at all. The entire result comes from 18 examples that disagreed, and 10 of those went your way.

Ten heads out of eighteen tosses. Nobody calls a coin bent on that.

The bootstrap, in five steps

That coin argument is the intuition. The bootstrap is the mechanical version of it, and it works on any score, not just right and wrong.

You would love to fetch a fresh 100 tickets and measure again, ten thousand times over. You cannot. So you fake it, using only the data you already have.

1. Write down one number per example: the fine-tune’s score minus the base model’s score on that same example. Here that gives 10 numbers of plus 1, 8 numbers of minus 1, and 82 zeros.
2. Draw 100 numbers at random from that pile of 100, putting each one back before the next draw. Duplicates are the whole point. Some examples get picked three times and some not at all, which is exactly what a different sample of reality would have looked like.
3. Average that fake sample. Write the average down.
4. Repeat steps 2 and 3 ten thousand times. You now have 10,000 plausible answers to the question “what could this delta have been”.
5. Sort those 10,000 averages. Discard the lowest 250 and the highest 250. What remains is the middle 95 percent, and that range is your 95 percent confidence interval.

Run it on the numbers above and the interval comes out at roughly minus 6 points to plus 10 points. The exact ends wobble slightly each time, because the resampling is random. The important part does not wobble: the range contains zero. There are plenty of plausible redraws in which the fine-tune is worse than the base model. Your 2-point gain is noise wearing a suit.

So the rule is: decide on the lower end of the interval, never on the average. If the lower end is below zero, you have not shown anything yet.

What actually fixes it

More examples. Keep the same rates and run the same evaluation on 2,000 examples instead of 100. Now 200 examples improve, 160 get worse, and the rest agree. The delta is still 2 points. The interval is now roughly plus 0.1 to plus 3.9 points.

It clears zero, and only just. That is the honest answer to “how big should my evaluation set be”. Not a round number somebody told you. Big enough that the difference you care about survives this test.

import numpy as np

def bootstrap_ci(base_scores, tuned_scores, n_boot=10000, alpha=0.05):
    diffs = np.asarray(tuned_scores) - np.asarray(base_scores)  # paired, per example
    n = len(diffs)
    boot_means = np.array([
        np.random.choice(diffs, size=n, replace=True).mean()
        for _ in range(n_boot)
    ])
    lo, hi = np.percentile(boot_means, [100 * alpha / 2, 100 * (1 - alpha / 2)])
    return diffs.mean(), lo, hi

mean_delta, lo, hi = bootstrap_ci(base_scores, tuned_scores)
print(f"delta={mean_delta:.3f}  95% CI=[{lo:.3f}, {hi:.3f}]")
# ship only if lo > 0

Line by line, assuming you have never used numpy.

• import numpy as np brings in numpy, the standard Python library for working on whole lists of numbers at once. The as np part is just a short nickname.
• base_scores and tuned_scores are two ordinary lists of per-example scores. They must be in the same order and cover the same examples, because the comparison is paired.
• n_boot=10000 is how many fake samples to draw. alpha=0.05 means you want a 95 percent interval, since 1 minus 0.05 is 0.95.
• np.asarray(...) - np.asarray(...) turns both lists into numpy arrays and subtracts them element by element. The result is step 1: one difference per example.
• n = len(diffs) is how many examples there are, so each fake sample is the same size as the real one.
• np.random.choice(diffs, size=n, replace=True) is step 2. It draws n values from diffs, and replace=True is what puts each value back before the next draw.
• .mean() is step 3, the average of that one fake sample.
• The square brackets with for _ in range(n_boot) around it are ordinary Python. They repeat the draw 10,000 times and collect the results. The underscore just means the loop counter is not used.
• np.percentile(boot_means, [2.5, 97.5]) is step 5, written as arithmetic. It returns the value that 2.5 percent of the averages fall below, and the value that 97.5 percent fall below. Those two are the ends of the interval.
• The f"..." line is a Python f-string. {lo:.3f} prints that number to three decimal places.

That gives you three rules that make a holdout earn its keep, and none of them is optional.

Climb the evaluation ladder and stop at the right rung

Evaluation is not one thing. It is a ladder, running from cheap and shallow up to expensive and true. Each rung measures something the rung below it cannot see.

Rung one, perplexity. Run the model over held-out text, take the average loss, and undo the logarithm. What comes out is roughly how many equally likely options the model felt it was choosing between at each token. A perplexity of 10 means it was about as unsure as someone picking between 10 options. Lower is better. It costs almost nothing, so you can run it at every checkpoint, and it catches gross damage quickly. It says nothing whatsoever about whether the model is good at your task.

Rung two, task metrics. Concrete, countable things. Did the extracted field match the expected value? Did the output parse as valid JSON? Was the classification correct? These are cheap and objective. One family of them deserves caution: overlap scores, which count how many words an answer shares with a reference answer. An answer sharing few words can still be perfect, and one sharing many can still be wrong, so overlap tracks human taste weakly.

Rung three, a model as judge. A strong model reads the answers and scores them against a written guide. The next section is entirely about this rung.

Rung four, human review. The ground truth, and the only rung that is not a proxy for something else. It is slow and expensive, and it needs written guidelines, or two reviewers will score the same answer differently and you will learn nothing.

The craft is running the cheap rungs constantly and the expensive rungs at milestones. The mistake is mistaking a low rung for the whole picture.

Use a model as a judge without being fooled by it

Here is the mechanism, because the phrase gets used more often than it gets explained. You take one prompt and two answers to it, usually one from the base model and one from your fine-tune. You hand both to a strong model, along with a rubric: a short written scoring guide, such as “score 1 to 5 on whether this resolves the customer’s problem, and 1 to 5 on tone”. The judge returns a score or picks a winner.

Why this caught on is worth stating precisely. Zheng and colleagues measured how often a strong judge picks the same winner a human picks, on the same pairs, and found agreement above 80 percent. Two humans agree with each other about that often as well. So the judge is roughly as consistent as a second reviewer, at a tiny fraction of the cost and time. That is genuinely useful.

It is also an instrument, and instruments have known errors. This one has three, and all three are easy to trip over.

• Position bias. The judge tends to prefer whichever answer it reads first, whatever the answers say. Always putting your fine-tune second quietly costs it points in every comparison you ever run.
• Verbosity bias. The judge tends to prefer the longer answer. A model that learned to pad will collect points it has not earned.
• Self-preference bias. The judge tends to prefer text written in its own style, which is a problem when the judge and one of the contestants come from the same family.

Each has a defence, and each defence is cheap. For position, score every pair in both orders and average the two results, or randomise the order per comparison. For verbosity, compare at matched output lengths, so quality is being read rather than word count. For self-preference and for drift generally, pin the exact judge model and the exact rubric version, then record both alongside the score. Otherwise next quarter’s improvement may just be a judge upgrade in disguise.

One more habit. Before you trust the judge on a thousand pairs, score fifty of them by hand and compare. If you and the judge disagree often, the rubric is usually the problem rather than the model. Treat the judge like any other measuring instrument in an eval harness: calibrate it first, then read it.

Catch the six failures a training curve cannot show

This is the spine of the part. Each failure below can sit happily beside a beautiful loss curve, and several can sit beside a strong headline metric too.

That pattern is not a coincidence. Training cannot surface these, because training is busy optimising the very number that hides them.

1. Overfitting

Symptom. Training loss keeps falling while performance on unseen data stalls or gets worse.

Cause. The model has started memorising your examples instead of learning the pattern behind them. On a set of ten thousand examples this happens quickly, often within two or three passes.

Check. You already have this one for free. Watch the gap between the training curve and the validation curve. While both fall, the model is generalising. The moment validation turns up while training keeps falling, you have crossed into memorisation. Keep the checkpoint from the low point, not the last one. Fewer passes over the data, a lower learning rate, more and more varied data, and regularisation all push the crossing point later.

2. Catastrophic forgetting

Symptom. The model is better at your task and quietly worse at things it used to handle: arithmetic, following a formatting instruction, refusing a request it should refuse.

Cause. This one is worth understanding properly, because it is the failure the whole series has been building toward. Nothing inside the model is labelled. There is no arithmetic module and no politeness module. The same 1.54 billion numbers do every job at once. Training pushes those numbers toward writing good refund replies, and some of the numbers it moved were also doing the carrying in a column of addition. Your loss only ever grades refund replies, so nothing objects.

Two things make it worse. Luo and colleagues found forgetting increasing with model size across the 1B to 7B range they studied. And running supervised fine-tuning followed by preference tuning induces it on its own, a pattern usually called the alignment tax.

Two further results are worth holding while you design the check. Allen-Zhu and Li found that a model can hold a fact and still fail to answer a question about it, depending on how varied the phrasings were when it first met that fact. Their follow-up put a rough ceiling on how much a model can store at all, near 2 bits of knowledge per parameter. Both say the same thing to anyone measuring a fine-tune: what you installed is mostly habit rather than new knowledge, so testing only your own task’s phrasing measures the smallest part of what changed.

Check. Score the same general-capability tests on the base model and on your candidate, with an identical setup, and compare. Three public tests are the standard probes, and it helps to know what each one is:

• MMLU is multiple-choice exam questions across many school and university subjects. It probes general knowledge.
• GSM8K is grade-school maths word problems. The answer is a number, so it is either right or it is not.
• IFEval is instructions with checkable constraints, such as “answer in exactly three bullet points”. A short program can verify compliance, with no judge involved.

The community tool for running them is EleutherAI’s lm-evaluation-harness. One command scores all three:

lm_eval --model hf \
        --model_args pretrained=./qwen15b-merged \
        --tasks mmlu,gsm8k,ifeval \
        --batch_size auto

Every part of that command, for someone meeting it for the first time:

• lm_eval is the command the harness installs. It downloads each test, runs your model over it and prints the scores.
• --model hf says the model is a local Hugging Face model on disk, rather than something behind an API.
• --model_args pretrained=./qwen15b-merged is the folder holding it. The word merged matters. If you trained a LoRA adapter, fold it into the base weights first, so the tool sees one plain model. Part 11 covers merging.
• --tasks mmlu,gsm8k,ifeval is the list of tests, comma separated.
• --batch_size auto lets the harness pick how many examples to push through at once, based on what fits on your card.
• The backslashes at the end of each line are shell syntax for “this command continues below”. Delete them and put it on one line if you prefer.

Run it twice. Once against the base checkpoint, once against the fine-tune. Then compare. Suppose, as an illustration, your ticket score rises 12 points while maths falls 6 and instruction following falls 3. That model has not improved. It has traded, and you now have to decide consciously whether the trade is one you want. Pin your library versions before you build any of this into a pipeline, because harness task names and defaults do move between releases.

Four mitigations, in rough order of how much they help. Prefer LoRA or QLoRA, because a frozen base forgets far less, and that is a large part of why these methods dominate. Lower the learning rate. Use fewer passes over the data, or a smaller adapter. And mix 5 to 10 percent general instruction data into your training set, so the model keeps rehearsing what it already knew.

3. Reward hacking

Symptom. The headline metric climbs steadily. You read twenty outputs and they are worse.

Cause. Whenever you optimise against a stand-in for quality, the model can find ways to score well without being good. The stand-in might be a reward model, which is a separate model trained to score answers, or a model judge, or a keyword-matching metric. Flattery, padding, and exploiting a gap in the rubric all raise the score. Part 13 covers where this starts.

Check. Partly manual, unavoidably. Read the outputs. Metric up plus eyeball bad is the signature, and there is no automated substitute for the eyeball half.

4. Length bias

Symptom. The tuned model waffles. Answers get longer and say no more.

Cause. Human annotators and model judges both tend to rate longer answers as better. If your preference data carries that habit, the model learns that length is quality.

Check. Run a length-controlled comparison. Compare quality only between answers of similar length, rather than on raw preference. If the win rate collapses once lengths are matched, the gain was padding.

5. Mode collapse

Symptom. Every answer starts to sound the same. The same opening, the same three-part structure, the same closing sentence.

Cause. Tuning too hard, for too long, crushes the variety out of the model’s answers. It settles into one safe template.

Check. Read many outputs, never a single sample. One answer always looks fine. Fifty answers reveal the template. Counting how often the same opening phrase appears across a hundred prompts takes ten minutes and finds it immediately.

6. Sycophancy

Symptom. The model agrees with the user. Even when the user is wrong.

Cause. Preference data collected from people rewards agreement, because being agreed with feels good in the moment and gets the thumbs up.

Check. Write prompts that assert a false premise, then see whether the model holds its ground. “This invoice totals 240 pounds, so why did you refund 260?” when the invoice does not total 240. A model that goes along with the premise has learned to please rather than to help. This is the same probing discipline as testing guardrails.

Know when a measure has stopped measuring

Pick any number and push on it hard enough and it stops telling you what it used to tell you. A school judged on pass rates starts teaching the exam. A support team judged on tickets closed starts closing tickets. The number rises. The thing the number stood for does not.

That has a name: Goodhart’s law. When a measure becomes a target, it stops being a good measure.

You have now met three faces of it in one article. Contamination is the benchmark being gamed by the pretraining data. Reward hacking is the model gaming your proxy. And there is a third, quieter one: you, tuning until the number rises. Each of them is the same phenomenon. A proxy under optimisation pressure drifts away from the thing it was meant to stand for. Loss is a proxy for capability. A benchmark is a proxy for skill. A reward model is a proxy for human judgement.

This is not a bug you fix once. It is a permanent condition you manage, and the defences are structural. Hold out fresh evaluations the model has never been optimised against. Rotate them over time. Combine several metrics, so gaming any one of them does not win. Keep a person in the loop on flagged cases. An evaluation you have been optimising against for six months is no longer measuring what it measured on day one.

Measure the thing your users actually need

General benchmarks tell you that you did not lose general competence. They tell you nothing about whether the model is good at your job. The evaluation that decides whether to ship is always built from your own data. Match the metric to the shape of the output.

Two of those terms need unpacking. A confusion matrix is a grid: real category down the side, predicted category across the top, counts in the cells. It shows which categories the model mixes up, which is far more useful than one accuracy number. Turning refund tickets into billing tickets is a different problem from turning both into nonsense, and one number cannot tell you which you have.

The second is worth a rule. Classification is the easiest thing in the world to evaluate honestly, because correctness is objective. If your task can be reshaped from open generation into a choice among a fixed set of valid answers, do it. You get a hard metric and a confusion matrix for free.

Two habits then make any task evaluation trustworthy. Build a regression suite, a fixed set of cases run against every checkpoint, so a fix in one area that breaks another is caught the same day. And run the evaluation under production conditions: the same prompt template, the same context assembly, the same routing. An evaluation that does not mirror production is measuring a model you will not deploy, which is exactly the trap described in the playbook for getting an LLM demo into production.

Run evaluation as a loop that starts before training

All of this assembles into one loop, and the order matters more than any individual metric.

Define what success means, and the number that clears the bar, before you train. Do it afterwards and you will rationalise whatever number you happen to get. Everyone does. Writing the threshold down first is the only defence.

Then baseline the base model on the same evaluation, so every result is a difference rather than a bare score. Train. Run the ladder. Compare the delta with its confidence interval. Read the failures. Ship or go round again.

Two pieces of plumbing make the loop stick. Wire the evaluation into your build, so every checkpoint is gated automatically against the thresholds you already committed to. And send a small slice of real traffic to the new model before a full rollout, so production gets a vote before it gets the whole load.

One framing worth carrying past this series. For anything client-facing or regulated, the evaluation is the evidence. A fixed holdout, a versioned rubric, a pinned judge, a base-versus-tuned delta with a confidence interval, and a stability check across runs is what turns a claim about your model into something you can defend. A claim without that artefact is just an assertion.

That closes the arc. You can now size a run before you start it, pick a base model, build the bench, run supervised fine-tuning end to end, fix the data that actually drives it, fit a large model on a small card with LoRA and QLoRA, teach judgement with preference tuning, split a job across cards when one is not enough, and prove whether any of it worked. None of those decisions rests on a black box, because every layer under them was built up in the parts before this one.

Where next. The training side is done, and the serving side is a separate discipline with its own arithmetic: the complete path an LLM takes to answer a question picks the story up at the moment your checkpoint goes into production. Go and fine-tune something, then measure it properly.