learn-ml-strategy-with-phoebe / Practitioner session 1 of 10
Learn ML Strategy with Phoebe · Practitioner session 1 of 10

Dev and test sets that tell the truth

Every number in the next nine sessions is measured on the dev set. If the dev set is drawn from the wrong traffic, every decision made on it is a decision about the wrong problem. If it is too small, improvements you actually made will be invisible and noise will look like progress. Lanternfish's first dev set was 400 daytime photos scraped from the web, for a classifier that runs on night infrared trail cameras. Tonight you build the one that replaces it, and learn the two questions that decide whether a dev set can be trusted at all.

🟢 Foundational Data scientists and ML engineers Anyone who has argued about whether a model got better 45 min
0-3 · Welcome 3-22 · Distribution and size 22-40 · Build-along: the Lanternfish dev set 40-45 · Q&A
Part 0

How the practitioner track works

Ten sessions, 45 minutes each, one decision: what to work on next for one classifier. Tonight the dev set it is all measured on. Then the first system in session 2, the error-analysis spreadsheet in 3, the Eyeball and Blackbox halves in 4, bias and variance in 5, learning curves in 6, data mismatch in 7, the Optimization Verification test in 8, error analysis by parts in 9, and in session 10 the whole decision is scored on a bench. The classifier throughout belongs to Lanternfish, a wildlife-camera startup whose species model is wrong on 12 percent of its dev set and whose team has five opinions about why.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. A dev set drawn from the distribution you will be judged on, not the one that was convenient. A size chosen from the smallest improvement you need to see, with the arithmetic that says 100 examples cannot detect a 0.1 percent change and 10,000 can. And the rule that saves the most months: when the dev set stops reflecting what you care about, change the dev set, not your opinion of the model.

LiveWhy the dev set comes before the model3 min▶

A machine learning project is a loop: try an idea, measure it on the dev set, keep it or drop it. The dev set is the only instrument in that loop. A team with a good dev set and a mediocre model will iterate its way to a good model; a team with a bad dev set and a brilliant model will iterate its way somewhere it did not mean to go, because every reading was of the wrong thing.

That is why the discipline starts here rather than with architecture. The two properties that make a dev set trustworthy are cheap to check and expensive to skip: it has to come from the distribution you will be judged on, and it has to be large enough to show the size of improvement you are going to be making.

Real world

Lanternfish's classifier scored 94 percent on its first dev set and 78 percent in the field. The dev set was 400 web photos, daytime, centred, sharp. The cameras produce night infrared frames with animals half out of shot. Every one of the six model changes accepted in the first quarter had been accepted because it moved the wrong number.

Self-studyThe vocabulary for the whole track4 min▶
  • Training set - what the model learns from. Can come from anywhere, as long as you know it did.
  • Dev set - what you tune and choose on. Must come from the distribution you care about. Also called the validation or hold-out set.
  • Test set - what you report on, once, at the end. Same distribution as the dev set, never tuned on.
  • Eyeball dev set - the part of the dev set you look at by hand, about 10 percent. Session 4.
  • Blackbox dev set - the rest, which you never look at, so it can still tell you the truth after you have tuned to what you saw. Session 4.
  • Single-number evaluation metric - one number the team optimises. Two numbers is a debate; one is a decision. Leader session 2.
  • Error analysis - looking at misclassified dev examples and categorising them, so the next project is chosen by ceiling rather than by interest. Session 3.
Part 1 · covers the distribution rule and its consequences

Dev and test come from the traffic you will be judged on

9 min live

The rule is one sentence: choose dev and test sets to reflect the data you expect to get in the future and want to do well on. The training set is allowed to be different; the dev and test sets are not. Almost every violation comes from convenience, and almost every violation is discovered in production.

Where the data comes from Web photos daytime, sharp, centred Camera archive last year, 40 cameras Live camera traffic what the field will judge What each set may draw from Training set anything, if you know what Dev set live traffic only, 1,000+ Test set live traffic only, touched once What went wrong last quarter dev set: 400 web photos dev accuracy: 94 percent field accuracy: 78 percent six model changes accepted, each because it moved a number about the wrong data the model was never the problem Training may come from anywhere. Dev and test come only from what the field will judge.
🔍 Click to zoom - three sources, and which sets are allowed to draw from each.
LiveTraining can differ, dev and test cannot3 min▶

The asymmetry surprises people. Training data is allowed to come from anywhere useful: web photos, a public dataset, last year's archive, synthetic augmentation. More of it usually helps, and session 7 covers what to do when it differs from production. Dev and test data are not allowed that freedom, because they are the definition of "doing well". A dev set from the wrong distribution does not measure your model badly; it measures a different problem precisely.

The practical test is a question: if the model scored perfectly on this dev set, would the field be happy? For Lanternfish's web-photo dev set the answer was obviously no, and nobody had asked.

LiveDev and test come from the same distribution as each other3 min▶

A second, quieter rule: the dev set and the test set must match each other. If the team tunes for months on a dev set from cameras in one region and the test set is cameras from another, the final number will disappoint and nobody will know which of two things happened: the model overfit the dev set, or the test set is simply a different problem. Drawing both from one pool and splitting at random removes the second explanation, so a gap between dev and test means exactly one thing.

Real world

Lanternfish's second attempt built a dev set from the northern cameras because that archive was labelled, and a test set from the southern ones because those were the paying customer. Dev went to 91, test came in at 84, and the team spent three weeks arguing about overfitting before anybody compared the two distributions. The model was fine. The sets were from different forests.

Self-studyWhen to change the dev set4 min▶

Three signals that the dev set, not the model, is the thing to change:

  • The field disagrees with the dev set. Users or downstream systems complain about a model the dev set says is good. The dev set no longer reflects the traffic. Rebuild it from current production data.
  • The model has overfit the dev set. Dev keeps improving, test does not. Too many decisions were made on the same examples. Get a fresh dev set, and adopt the Eyeball and Blackbox split from session 4 so it cannot happen again.
  • The metric rewards the wrong thing. A classifier that ranks a rare protected species below a common one might score well on accuracy and badly on the job. That is a metric change, leader session 2, and it usually comes with a dev set change because the examples you weight have to be present.

What is not a signal: a number you dislike. A dev set that says the model is worse than you hoped is doing its job.

Part 2 · covers dev set size

Large enough to see the improvement you are going to make

8 min live

The size question has a clean answer once it is asked properly. Not "how much data do we have" but "how small an improvement do we need to detect". A dev set of 100 examples cannot tell a 90.0 percent classifier from a 90.1 percent one: the difference is a tenth of an example. Dev sets of 1,000 to 10,000 are common for exactly this reason, and at 10,000 a 0.1 percent improvement becomes visible.

100 examples 1 percent = 1 example 0.1 percent = a tenth of one cannot see 90.0 vs 90.1 noise the size of the change you are trying to make Lanternfish's first set: 400 1,000 examples 1 percent = 10 examples 0.1 percent = 1 example sees a 1 point change the common size, and enough for a 12 percent error rate Lanternfish's new set: 1,000 10,000 examples 1 percent = 100 examples 0.1 percent = 10 examples sees a 0.1 point change for mature systems where tenths of a point are the game search, ads, recommendations Size the dev set from the smallest improvement you need to see, not from the data you happen to have.
🔍 Click to zoom - what each size can and cannot see.
LiveCounting the examples an improvement is worth4 min▶

The arithmetic is one multiplication. An improvement of one point on a dev set of 1,000 is ten examples going from wrong to right. On a set of 100 it is one example, and one example flips for reasons that have nothing to do with the model. So before choosing a size, write down the smallest improvement a project is expected to deliver, and make that improvement worth at least a handful of examples.

For Lanternfish at 12 percent error, the projects in session 3 promise between one and three points each. A thousand examples makes a one-point change ten examples and a three-point change thirty, both clearly visible. Ten thousand would be nicer and would cost ten times the labelling; a thousand is enough for the decisions this team is about to make.

★ Try it now -- how many examples does each candidate improvement move, at each dev set size? with sizes as (select unnest(array[100, 400, 1000, 10000]) as n), gains as (select unnest(array[0.001, 0.01, 0.03]) as gain) select n, gain, round(n * gain, 1) as examples_moved, case when n * gain < 3 then 'invisible' else 'visible' end as verdict from sizes cross join gains order by n, gain;
Self-studyThe test set, and how not to spend it3 min▶
  • Size it to the confidence you need in the final number, not to a fraction of the data. The old 70/30 habit made sense when data was scarce; with a million examples, 30 percent held out is waste.
  • Touch it once. Every decision made on the test set turns it into another dev set. If the test number is disappointing, the answer is a new test set, not a second look.
  • Keep it from the same pool as the dev set, so a dev-to-test gap can only mean one thing.
  • Report it with the dev number beside it. A test result on its own hides whether the team overfit the dev set, which is the failure session 4 is built to prevent.
Build-along

Build the Lanternfish dev set

★ 18 min · everyone builds

Bring a classifier you own and the traffic it serves. If you have nothing to hand, use the Lanternfish frame log: 40 cameras, eleven months, 2.1 million frames, of which 38,000 have a species label from the field team.

Write down what the field will judge. One sentence: "night and day frames from the deployed cameras, as they arrive, including empty frames". Everything that follows samples from that sentence and nothing else.

Pull from production, not from the archive of convenience. Sample labelled frames from the last three months of live cameras, stratified by camera and by day-versus-night in the proportions the cameras actually produce. Lanternfish's cameras are 61 percent night frames; the old dev set was 3 percent.

Decide the smallest improvement you need to see. The candidate projects promise one to three points. Make one point worth at least ten examples: 1,000 for the dev set. Draw a second 1,000 from the same pool for the test set and lock it.

Check the two sets against each other. Camera mix, night share, species mix. If any differs by more than a few points, the split was not random enough. Redraw before anybody trains anything.

Score the current model on the new dev set. Lanternfish's 94 percent becomes 88 percent: 120 of 1,000 wrong. Write that number down. It is the baseline every later session measures against, and it is the first honest number the project has had.

Split the dev set in half and put one half away. Five hundred you will look at, five hundred you will not. Session 4 explains why, and the bench in session 10 shows what happens to a team that skips this step.

★ The dev set, drawn from what the field will judge with pool as ( select frame_id, camera_id, is_night, species_label from cams.frames where labelled_at is not null and captured_at >= dateadd('month', -3, current_date) ), draw as ( select *, row_number() over (order by hash(frame_id)) as rn from pool ) select frame_id, camera_id, is_night, species_label, case when rn <= 1000 then 'dev' else 'test' end as split, case when rn <= 1000 and rn % 2 = 0 then 'eyeball' when rn <= 1000 then 'blackbox' end as dev_half from draw where rn <= 2000; -- then compare camera mix and night share across dev and test before training anything

The number went down and that is the good outcome. 94 to 88 is not the model getting worse. It is the instrument getting honest, and every project funded from here is funded against a real problem.

Homework

Before session 2

30 min
  1. For one model you own, write the one-sentence description of what the field will judge, and check whether the current dev set matches it. Most do not.
  2. Count your dev set and compute what one point of improvement is worth in examples. If it is under ten, the set is too small for the decisions being made on it.
  3. Compare your dev and test sets on three obvious dimensions. Write down any gap over a few points.
  4. Find out how many times the test set has been looked at this year. Be honest. If the answer is more than once, it is a dev set now.
Check yourself

Three questions

Lanternfish's classifier scored 94 percent on a dev set of web photos and 78 percent in the field. What was wrong?

Dev and test sets have to come from the traffic you will be judged on. A dev set from the wrong distribution does not measure the model badly; it measures the wrong thing well.

Classifier A is 90.0 percent accurate and classifier B is 90.1 percent. Why can a dev set of 100 examples not tell them apart?

Size the dev set from the smallest improvement you need to see. At 10,000 examples the same 0.1 percent is ten examples and becomes visible.

Dev accuracy climbed to 91 percent over a quarter; the test set came in at 84. Dev and test were drawn from different camera regions. What can you conclude?

Dev and test must come from the same pool so a gap between them can only mean one thing. Two forests, two explanations, three weeks of argument.

Sources covered

What tonight maps to

✓ Machine Learning Yearning, chapters 5 to 7 Your development and test sets; same distribution; how large they need to be, with the 100-versus-10,000 arithmetic quoted from the text.
✓ Machine Learning Yearning, chapter 11 When to change dev/test sets and metrics: the three signals.
◐ Machine Learning Yearning, chapters 17 to 18 The Eyeball and Blackbox split is done tonight and explained in session 4.
◐ Choosing the metric Leader session 2. Tonight assumes accuracy and says so.
· Train/test split mechanics, cross-validation, leakage Not covered by design. The intro ML course owns the mechanics; this track owns the distribution and size decisions.

Session 1 cheat sheet

The dev set is the instrumentEvery decision in the loop is a reading of it. Fix the instrument before the model.
Dev and test: the field's distributionTraining may come from anywhere. Dev and test only from what you will be judged on.
Dev and test: the same poolSo a gap between them means exactly one thing.
Size from the improvementMake one point worth at least ten examples. 100 cannot see 0.1 percent; 10,000 can.
The test set is touched onceA second look makes it a dev set.
Change the dev set whenThe field disagrees, dev improves and test does not, or the metric rewards the wrong thing.
A number that goes down94 to 88 was the instrument getting honest, not the model getting worse.
Put half awayEyeball and Blackbox. Session 4 says why; session 10 shows the cost of skipping it.