Two halves, two jobs
The dev set has been doing two incompatible jobs at once. It is the material you read to work out what is broken, and it is the instrument that tells you whether a change helped. Reading it well makes it worse at measuring. Splitting it lets each half do one job properly: an Eyeball half you open, categorise and argue about, and a Blackbox half nobody looks at, ever, which is therefore still an independent measurement after six months of tuning.
★ What you walk out with tonight. Lanternfish's 1,000 dev images split into a read half and a sealed half, with both error rates written down: 64 errors in the Eyeball half at 12.8 percent, 56 in the Blackbox half at 11.2. A reporting habit that puts the two numbers next to each other on every run. And the shape of the failure they exist to catch, produced on purpose in a build-along: one half drops to 3.8 percent in two weeks while the other stays exactly where it was.
LiveWhy reading a set costs you the set3 min▶
An hour spent reading dev examples is an hour of fitting your own judgement to those examples. It happens without any bad faith and without anyone touching a training script. You notice that four of the night frames have a bright halo, so you add a halo check. You remember that two elk pictures had antlers cut off at the top, so you widen a crop. Each of those is a reasonable idea, and each of them was suggested by a specific image that lives in the set you are about to score on.
A dev set you have read hundreds of times is not an independent measurement any more. It is a set of examples your team has been optimising against with its eyes, and the number it produces is closer to a training score than to a hold-out score. The only defence that survives contact with a motivated team is a portion of data that nobody is allowed to open.
Lanternfish's team read all 120 errors in session 3. Two of them could recite frame ids. When a new preprocessing step landed the following week, the first thing anyone did was check the six night-blur frames they remembered by name. All six were fixed. The dev error moved by 0.4 points.
Self-studyThe vocabulary for tonight3 min▶
- Eyeball dev set - the part you look at with your eyes, categorise by hand, and use to decide what to work on. Its error rate is a diagnostic, not a report.
- Blackbox dev set - the part you never open. Used only to score models. Its error rate is the one you report.
- The gap - Eyeball error minus Blackbox error. Small and stable is healthy. Growing is a warning that the Eyeball half is being tuned to rather than learned from.
- Retiring a set - replacing an Eyeball half that has been read so many times its number no longer tracks the Blackbox half. It is normal, it is planned for, and it costs a labelling run.
- Test set - still separate from both halves, still touched once, still from the same pool as the dev set. Session 1 owns that decision.
"If you have a large dev set, split it into two subsets, only one of which you look at"
9 min liveThat is the title of chapter 17 of Machine Learning Yearning, and the mechanism it describes is one sentence long: "randomly select 10% of the dev set and place that into what we'll call an Eyeball dev set to remind ourselves that we are looking at it with our eyes". The remainder is the Blackbox dev set. You tune on it, you score on it, you never open it.
LiveWhy Lanternfish splits 500/500 rather than 10/904 min▶
Ten percent of 1,000 images is 100 images, and 100 images at a 12 percent error rate holds about twelve errors. Twelve errors cannot be categorised into five buckets in any way that supports a decision: a category with three errors and a category with two are indistinguishable, and the ceiling arithmetic from session 3 would be reading noise.
Chapter 15 gives the number to aim at from the other direction: "I usually create a spreadsheet and fill it out while looking through ~100 misclassified dev set images." What matters is how many errors the Eyeball half contains, not how many images. Half of Lanternfish's dev set holds 64 errors, which is close enough to a hundred to fill a spreadsheet with categories that separate. So the split is 500 and 500, and the ten percent rule is what you use when the dev set is large enough for ten percent to carry a hundred mistakes.
Read the sizing question in errors, not images. Chapter 18 asks how big the Eyeball and Blackbox dev sets should be, and the useful form of the question is: does the Eyeball half hold enough mistakes to categorise, and does the Blackbox half hold enough examples to detect the improvement you are trying to make?
LiveThe two halves do not start equal, and that is the point3 min▶
Lanternfish's split is random, and random placement of 120 errors across two halves of 500 left 64 in one and 56 in the other. That is a 1.6-point difference at the starting line, before anybody has done anything. Teams see it and want to rebalance until the halves match.
Do not. The whole value of the Blackbox half is that it is a fresh sample the team has no relationship with, and stratifying it to match the half you read hands back a little of the independence you just bought. The 1.6-point starting gap is a fact about sample size, and it is the yardstick for later: a gap that stays around 1.6 is two halves agreeing, and a gap that opens to 7.4 is something else entirely.
Self-studyDeterministic splits, and why hash beats random4 min▶
- A split that moves is not a split. If the assignment is redrawn every time the query runs, images drift between halves and the Blackbox half quietly acquires examples somebody read last month.
- Hash the stable id, not the row order.
hash(frame_id) % 2gives the same answer forever, survives a reload of the table, and needs no stored column. Row numbers change when the table changes. - Store the assignment anyway. A materialised
dev_halfcolumn means the split survives a change of database engine, which changes the hash function under you. - New images join by the same rule. When the dev set is topped up, hash decides the half. Nobody chooses which half a new image lands in, because whoever chooses will choose in a way that flatters something.
- Access control is cheaper than discipline. A view that exposes only the Eyeball half is a better guarantee than an agreement not to look.
How to tell you have overfit the half you looked at
8 min liveTwo numbers, reported side by side, on every run. The Blackbox rate is the result. The Eyeball rate sits next to it as a control. When they move together, the work is real. When the Eyeball rate improves and the Blackbox rate does not, the team has been fixing the examples it read rather than the problem those examples are evidence of, and the size of the gap is the size of the illusion.
LiveThe demonstration, and why it is honest about being a construction4 min▶
Two weeks of hand rules written against the 64 errors in Lanternfish's Eyeball half take that half from 12.8 percent to 3.8 percent. The Blackbox half stays at 11.2 percent, exactly where it started. The reported improvement, if the team reported the number it had been watching, would be a 70 percent reduction in error. The real improvement, measured on data nobody touched, is zero.
This is a construction by definition and it is worth saying so plainly rather than dressing it up as a study of human behaviour. A rule keyed to a specific frame id fixes that frame and nothing else; there is no uncertainty in the arithmetic and nothing about anybody's psychology is being modelled. The construction earns its place because it draws the shape at full contrast: the two lines separating is what a memorised dev set looks like on a chart.
Nobody writes rules keyed to frame ids on purpose. What happens instead is slower and much harder to see: a threshold nudged because two remembered images sat just the wrong side of it, a class weight raised because the elk errors felt more numerous than they were, an augmentation chosen to match the four blurry frames that stuck in the mind. The same gap opens over three months instead of two weeks, one decision at a time, and every decision is defensible on its own.
LiveReading a gap, and what different sizes mean3 min▶
| What you see | What it means | What to do |
|---|---|---|
| Both halves fall together | The change fixed the problem the examples were evidence of | Keep it, report the Blackbox number |
| Gap stable near its starting value | Sampling noise, nothing more | Nothing. Lanternfish starts at 1.6 points |
| Eyeball falls, Blackbox flat | The change fixed the examples, not the problem | Do not ship it. Ask what the rule keyed on |
| Gap grows slowly over months | Ordinary tuning has been fitting the read half | Retire the Eyeball half and draw a fresh one |
| Blackbox falls, Eyeball flat | Usually a small-sample accident, occasionally a category the Eyeball half is short of | Check the category counts before celebrating |
Self-studyRetiring an Eyeball set, and the trade when the dev set is small4 min▶
An Eyeball set is consumable. It is read, argued over, and eventually so familiar that its number stops tracking the sealed half. Plan for that instead of being surprised by it.
- The retirement signal is a gap that opens and will not close. Two model changes in a row that move the Eyeball rate and leave the Blackbox rate alone is the point at which the read half has stopped being a measurement of anything.
- Retirement means fresh data, not a reshuffle. Moving images from the Blackbox half into the Eyeball half spends the only clean data you have. Draw new labelled examples from current production traffic, exactly as session 1 drew the first set.
- Budget for it. A team reading errors seriously will burn an Eyeball half every few months. That is a recurring labelling cost, and it is cheaper than a quarter of decisions made on a memorised set.
- When the dev set is small, the trade is real and there is no clever way out. With 200 images, a half holds about a dozen errors. Categorise on the whole set, accept that the measurement is compromised, and buy back honesty with a test set that stays genuinely untouched.
- A small Blackbox half has its own limit. Session 1's arithmetic still applies: 500 examples makes a one-point improvement five examples, which is visible; a tenth of a point is half an example, which is not.
- Two halves cost nothing to keep. The split is a column. The discipline is a report with two columns instead of one, and neither is a project.
Split it, confirm it, then break it on purpose
★ 20 min · everyone buildsBring the dev set from session 1 and the error table from session 3. If you have nothing to hand, the Lanternfish tables are cams.dev_set with 1,000 rows and cams.dev_errors with the 120 misclassified frames and their categories.
Assign the halves deterministically, if session 1 did not. Hash the frame id so the assignment is the same on every run, on every machine, forever. Materialise it as a column so a change of engine cannot silently reshuffle who is in which half.
Count errors and the error rate per half, and confirm the two numbers. You are looking for 64 errors in the Eyeball half at 12.8 percent and 56 in the Blackbox half at 11.2 percent. If your split gives 60 and 60, the hash is doing something orderly rather than random, and that is worth ten minutes to find.
Write both numbers into a results table with a run label. One row per model version, with both rates and the gap as a stored column. This is the artifact that makes the failure visible three months from now, when nobody remembers what the gap used to be.
Simulate the trap. Write a rule keyed to a list of specific Eyeball frame ids, apply it, recompute both halves, and watch the gap open to 7.4 points. Doing it on purpose once is what makes it recognisable when it arrives by accident.
Write the reporting rule down where the team will hit it. Any run that reports one error rate is incomplete. Put the check in the same place the number is produced, so the two-column habit does not depend on anybody remembering.
Restrict access to the Blackbox half. A view that returns only Eyeball rows, and a scoring job that reads the sealed half without displaying an image. An agreement not to look is not a control.
Make the report refuse to print one number. The cheapest durable version of tonight is a scoring script that raises if either half is missing, and prints eyeball / blackbox / gap as a single line. A convention survives about six weeks; a script that fails survives the team.
Before session 5
30 min- Split one dev set you own into Eyeball and Blackbox halves by hash of a stable id, and record both error rates and the starting gap. That gap is your baseline for every comparison after this.
- Count the errors in your Eyeball half. Under about fifty and the categorisation from session 3 is reading noise; say so out loud rather than working around it.
- Go back through the last five model changes your team accepted. For each, write down whether anybody could tell you today which half improved. If the answer is no five times, that is the finding.
- Add the two-column report to wherever your scoring runs, and make it fail rather than warn when a half is missing.
- Ask how many times each error image has been opened this quarter. Any image somebody can describe from memory belongs in the Eyeball half or nowhere.
Three questions
After two weeks of work the Eyeball half reads 3.8 percent and the Blackbox half reads 11.2 percent, unchanged. What does the 7.4-point gap tell you?
A rule written against sixty specific images fixes those sixty images. The half nobody opened is the only reading left that has not been tuned against, and it says nothing improved.
Lanternfish has 1,000 dev images at 12 percent error. Why is the Eyeball half 500 images rather than the 100 that chapter 17's ten percent would give?
Size the Eyeball half by the errors it contains, not the images. Ten percent is the right rule once ten percent carries roughly a hundred mistakes.
The two halves start at 12.8 and 11.2 percent. A teammate proposes rebalancing them so both start at 12.0. What is wrong with that?
Random placement of 120 errors leaves the halves uneven, and that unevenness is a fact about sample size. Write the starting gap down and compare against it.