The nine points that were never one thing
Training error 3, dev error 12, and session 5 called the nine-point gap variance. That reading assumed the two sets were the same kind of image. They are not: the training pool is mostly daytime web photos and the dev set is camera frames from the deployed cameras. Some of the nine points are a model that failed to generalise. The rest are a model that was never shown the problem. They fund completely different work.
★ What you walk out with tonight. The test for whether mixing distributions is the right call at all, rather than a habit. A training-dev set carved from the training pool, held out and never trained on, which turns four sets into four numbers. The three gaps between those numbers and what each one names. And the limit that keeps synthesis honest: a model that loves your simulated night frames may still fail on real ones.
LiveWhy the gap became unreadable3 min▶
The bias and variance arithmetic from session 5 rests on a quiet assumption: the training set and the dev set are drawn from the same distribution, so the only thing that changes between them is whether the model has seen the examples. Break that assumption and the dev minus training gap stops being a single quantity. It becomes a sum of two, and the sum is all you can see.
The two components pull in opposite directions when it comes to work. Variance says the model overfits, so buy more of the same kind of data, augment, or regularise. Mismatch says the model is fine on what it was shown and was shown the wrong thing, so change the input distribution. Running the variance playbook against a mismatch problem is the most common expensive mistake in this part of the discipline, and it is invisible without a fourth set.
Lanternfish spent six weeks on dropout, weight decay and heavier augmentation to close a nine-point gap that the team had labelled variance. Dev error moved from 12.4 to 12.0. When they finally carved a training-dev set, three of the nine points were variance and six were mismatch. Six weeks of correct technique aimed at a third of the problem.
Self-studyThe four sets, in one place3 min▶
- Training set - web photos plus camera frames, the mixed pool. The model learns from this and only this.
- Training-dev set - carved from the same mixed pool, in the same proportions, held out and never trained on. Same distribution as training, unseen like dev.
- Dev set - 1,000 camera frames, the target distribution. Tuned and chosen on.
- Test set - camera frames from the same pool as the dev set, reported once.
The training-dev set is the only one of the four that has no job in the field. It exists purely as an instrument: it holds the distribution constant while the seen-versus-unseen thing changes, so the difference it exposes can only be one of the two.
When you should train and test on different distributions
9 min liveChapter 36 of Machine Learning Yearning is titled "When you should train and test on different distributions", and the word that carries it is "should". Mixing is not a compromise you apologise for. It is often the right engineering call, and there is a test for when it is not.
LiveThe Lanternfish arithmetic for keeping the web photos3 min▶
Lanternfish has 200,000 labelled web photos, free, already in hand, daytime and sharp. It has 38,000 labelled camera frames, at 4 pounds each and about five months of field-crew time per 38,000. Keeping the distributions identical means training on 38,000 examples instead of 238,000, and throwing away six times the data to preserve a property that matters for measurement, not for learning.
That trade is almost always the wrong way round. Dev and test have to be camera frames because they define what doing well means. The training set has no such duty: its only job is to teach a model to recognise a deer. A deer in a daytime web photo is still a deer, and the low-level structure the model learns from 200,000 of them transfers to a night frame far better than nothing does.
The asymmetry, restated. Training data is chosen to teach. Dev and test data are chosen to measure. Optimising a training set for measurement properties is how a team ends up with a well-measured model that nobody trained properly.
LiveWhether to include inconsistent data at all3 min▶
Chapter 38 asks the question in the other direction: given data that does not match your target, should it be in the training set at all? The test is whether the extra data teaches something the target distribution shares.
- Web photos of deer, elk and foxes. Shared: what the animals look like, how bodies occlude, how many legs point which way. Include them.
- Satellite imagery of forests. Shared: nothing the classifier needs. Adding it spends capacity on a mapping the target never asks for. Leave it out.
- Web photos with a different label taxonomy. Shared content, incompatible labels. Fix the taxonomy first or the data teaches the model to be confidently wrong.
- Frames from a discontinued camera model. Shared: night infrared texture, framing, motion blur. Include, and expect it to be the most valuable non-target data you own.
Two things reduce the risk of a wrong call here. Weight the target-distribution examples more heavily than the off-distribution ones, so the model is pulled toward the thing it is judged on. And run the four numbers below before and after adding a source, because the training-dev set will show you what the new data did to generalisation and what it did to mismatch, separately.
Self-studyWhat mixing costs you, precisely3 min▶
One thing, and it is a measurement thing rather than a modelling one. With a single distribution, dev error minus training error has exactly one meaning. With two, it has two, superimposed:
- The model did not generalise. It fits examples it saw better than examples it did not, regardless of what kind they are. This is variance in the ordinary sense.
- The model never saw this kind of image. It generalises perfectly well within the world it was trained on, and the dev set is a different world. This is data mismatch.
Nothing in the two numbers separates them, and teams argue about which it is for months. The cost of mixing is one confounded reading, and the fix is one more held-out set rather than a smaller training pool.
The training-dev set separates the two
9 min liveHold out a slice of the training pool, in the same proportions as the pool, and never train on it. It is unseen like the dev set and it is web-photos-plus-camera-frames like the training set. Comparing against it holds one variable still at a time, and four numbers fall out where two used to be.
LiveCarving it, and the one rule that makes it work3 min▶
Take the mixed training pool and hold out a random slice before training. Two thousand examples is plenty for Lanternfish; the training-dev set only has to be big enough to measure a gap of a few points, and session 1's arithmetic applies unchanged.
The rule that makes it an instrument rather than decoration: it must have the same composition as the training set. Lanternfish's pool is 84 percent web photos and 16 percent camera frames, so the training-dev set must be too. A training-dev set accidentally enriched with camera frames drags the mismatch gap into the variance gap, and you get a confident number for the wrong split.
LiveReading the four numbers4 min▶
Four errors, three differences, and each difference isolates a single change:
- Human-level to training error is avoidable bias. Same as session 5, unchanged by any of this.
- Training to training-dev is variance. Same distribution on both sides, so the only thing that changed is whether the model saw the examples.
- Training-dev to dev is data mismatch. Both unseen, so the only thing that changed is which distribution the examples came from.
- Dev to test is dev overfit. Same distribution, both unseen by the model, so the only thing that changed is how many decisions you made while looking at the first one.
Lanternfish's four numbers are 3, 6, 12 and 12. Variance is 3, mismatch is 6, dev overfit is 0. The nine points session 5 attributed to variance were two thirds mismatch, and the six weeks of regularisation were aimed at the smaller third.
| Comparison | What is held still | What changes | Lanternfish | Name |
|---|---|---|---|---|
| Human-level to training | Nothing about data | Model versus person | 1 point | Avoidable bias |
| Training to training-dev | Distribution | Seen versus unseen | 3 points | Variance |
| Training-dev to dev | Unseen by the model | Distribution | 6 points | Data mismatch |
| Dev to test | Distribution and unseen | Decisions made on it | 0 points | Dev overfit |
LiveAddressing data mismatch4 min▶
Chapter 42 is titled "Addressing data mismatch" and its first instruction is not a technique. Look at examples from both sides and work out what actually differs. Not what you assume differs. Lanternfish's team expected the answer to be "night", and it partly was, but the bigger differences were motion blur from a triggered shutter and animals cropped by the frame edge, neither of which appears in a curated web photo.
Once you know what differs, there are two families of fix and they cost very different amounts:
- Make the training data more like the target. Synthesise the difference: convert web photos to infrared-like greyscale, add the sensor's noise profile, apply the motion blur the trigger produces, crop animals at the frame edge. Cheap, fast, and applies to all 200,000 photos at once.
- Collect more target data. More labelled camera frames, which is 4 pounds each and five months of field crew per 38,000. Expensive, slow, and unarguably the real distribution.
Do them in that order and measure between. Synthesis often takes a six-point mismatch to two, at which point the collection project is competing against a much smaller ceiling and may not be worth funding at all.
Self-studyThe honest limit of synthesis4 min▶
Artificial synthesis has a failure mode that looks exactly like success. Your synthesiser produces one narrow slice of the space of night frames, the model learns that slice thoroughly, the training-dev-to-dev gap closes, and the field number does not move. The model overfit your synthesis process, not the target distribution.
- The tell. A mismatch gap that closes far faster than the field error does. If synthesis takes mismatch from 6 to 1 and the deployed accuracy is unchanged, you synthesised something the cameras do not produce.
- A person cannot audit this by eye. Simulated night frames that look convincing to a human can be trivially separable to a model, which is exactly what lets it overfit them.
- Vary the synthesis parameters widely, and randomise them per example rather than fixing one noise level and one blur radius across 200,000 photos.
- Keep some real target data out of the synthesis loop entirely, so there is always a number that synthesis cannot have flattered.
- Report the field number next to the mismatch gap. Two numbers that move together is a fix; one that moves alone is a story about your synthesiser.
Separate variance from mismatch on your own model
★ 20 min · everyone buildsBring a model whose training data is not purely production traffic, which is most of them. If you have nothing to hand, use the Lanternfish pool: 200,000 web photos, 38,000 labelled camera frames, a 1,000-frame dev set with 120 wrong, and a 1,000-frame test set drawn from the same camera pool.
Carve the training-dev set before you train anything. Two thousand rows, drawn at random from the whole mixed pool, in the same source proportions as the pool. Write the split to a table so every future run uses the same one.
Verify the composition matches. Source mix, day-versus-night share, species mix. If the training-dev set is 20 percent camera frames and the pool is 16 percent, redraw. An enriched training-dev set moves mismatch into variance silently.
Train on the training split only, and produce four errors: training, training-dev, dev, test. Score training error on the training split itself, not on the pool including the held-out slice.
Compute the three gaps and name the dominant one. Training to training-dev is variance, training-dev to dev is mismatch, dev to test is dev overfit. Whichever is largest is the one the next sprint is about.
Pull ten examples from each side of the mismatch. Ten training-dev examples the model got right and ten dev examples it got wrong, side by side on one screen. Write down what actually differs, in plain words, before anybody proposes a fix.
Propose one synthesis step and one collection step, each with a cost. "Infrared greyscale plus sensor noise on all 200,000 web photos, two days of engineering" against "another 10,000 labelled camera frames, 40,000 pounds and six weeks". Both go in the backlog table with a ceiling, exactly like every other project.
Re-run the four numbers after the synthesis step. Mismatch is the number that should move. If variance moves instead, the synthesis changed the training distribution in a way you did not intend.
Look at the images before the histogram. A distribution comparison over the columns you already have can only find differences you already thought to record. The three most valuable differences on the Lanternfish list had no column, and twenty minutes of looking is what put them there.
Before session 8
40 min- Work out what share of your training data comes from the same source as your dev set. If it is under half, the gap you have been calling variance is mostly something else.
- Carve a training-dev set from your training pool and verify its composition matches. Persist the split so it is the same one next month.
- Produce the four numbers and the three gaps. Put them in the same place your team already reads the training and dev numbers, so the split becomes the default reading rather than an investigation.
- Look at ten examples from each side of your largest gap and write down what differs, in plain words, with no fix attached.
- Cost one synthesis step and one collection step against that difference. Take the cheaper one to whoever approves the sprint, with the mismatch gap it is aimed at written next to it.
Three questions
Training error 3, training-dev error 6, dev error 12. Which gap is data mismatch, and how much of it is there?
Each gap changes exactly one variable. Training to training-dev holds distribution still and changes seen versus unseen, so it is variance. Training-dev to dev holds unseen still and changes distribution, so it is mismatch.
Lanternfish has 200,000 free web photos and 38,000 camera frames at 4 pounds each. Why not train on camera frames only, so every set matches?
Dev and test have to be camera frames because they define doing well. The training set has no such duty. The measurement problem the mix creates is fixed with one more held-out set, not with a smaller training pool.
Synthesised night frames take the mismatch gap from 6 points to 1, and the deployed accuracy does not move. What happened?
Simulated frames that look convincing to a person can be trivially separable to a model. Vary the synthesis parameters per example, keep some real target data out of the loop, and always report the field number next to the mismatch gap.