What one error rate hides
A 12 percent dev error is not one thing. Part of it is error nobody could remove, because the images genuinely do not contain the answer. Part of it is the model failing to fit even the data it was shown. The rest appears only when the model meets data it has not seen. Those three parts have different causes, different fixes and different costs, and a single number keeps all three invisible.
★ What you walk out with tonight. Lanternfish's 12 percent split into unavoidable bias 2, avoidable bias 1 and variance 9, from three numbers you can measure this week. A human-level estimate from 50 hand-labelled dev images, which is the number that tells a hard problem apart from a weak model. And a check on the current sprint: which of the two project families it belongs to, and whether the decomposition agrees.
LiveThe three numbers, and where each comes from3 min▶
- Training error. The same metric, computed on the data the model learned from. Cheap: you already have the predictions. If it is high, the model cannot even fit what it was shown.
- Dev error. The Blackbox half from session 4, because after four sessions of reading errors it is the only honest reading left. Lanternfish: 12 percent.
- Human-level error. What a careful person gets wrong on the same images. Not free: it costs a labelling exercise. It is also the number that decides whether 3 percent training error is a triumph or an embarrassment.
Lanternfish's team argued for a month about whether to buy more labelled night frames or move to a larger backbone. Both cases were well made. The three numbers took an afternoon and said the model fits its training data almost as well as a human reads the images, and falls apart on data it has not seen. That is a data and generalisation problem, and the larger backbone was the wrong quarter.
Self-studyThe vocabulary, carefully3 min▶
- Optimal error rate - the error no model could remove, because the information is not in the input. A night frame with one ear visible does not say elk or deer, and no architecture will change that. Also called unavoidable bias.
- Avoidable bias - the gap between training error and the optimal error rate. Error the model is making on data it has already seen, which is therefore in principle removable.
- Variance - dev error minus training error. The part that appears only on unseen data.
- Human-level performance - the practical stand-in for the optimal error rate on perception tasks, and the reason chapter 33 exists.
- Orthogonalization - one knob per problem, so a change that moves two numbers at once cannot be reasoned about. Course material from Ng's Structuring Machine Learning Projects, not from the book.
"Bias and Variance: The two big sources of error"
9 min liveThat is chapter 20 of Machine Learning Yearning, and chapter 22 does the arithmetic on a worked example. A system with 30 percent dev error, on a task where the optimal error rate is 14 percent, has avoidable bias of 1 percent, "calculated as the difference between the training error and the optimal error rate". Variance is dev error minus training error: 15 points. The 30 percent that looked like a broken model turns out to be a hard task with a generalisation problem on top.
LiveTwo subtractions, in the right order4 min▶
Write the three numbers in a column: human-level 2, training 3, dev 12. Then read the gaps between adjacent rows. Training minus human-level is avoidable bias: 1 point. Dev minus training is variance: 9 points. What is left underneath, the human-level number itself, is unavoidable bias: 2 points. The three add back to the 12 you started with, which is the check that you did it right.
The order matters because it tells you which gap to attack. Lanternfish's model fits its training data to within one point of a careful human, so there is almost nothing to win by making it fit training data better. Nine of its twelve points appear only on data it has not seen. Whatever the team does next, if it does not close that gap it is not addressing 75 percent of the error.
LiveWhy the human-level estimate is not optional4 min▶
Chapter 33 asks why we compare to human-level performance, and the practical answer is that without it you cannot tell a hard problem from a weak model. A 3 percent training error is close to flawless on a task where a careful person gets 2 percent wrong. The identical 3 percent is a serious failure on a task where a person gets 0.1 percent wrong, because it means 29 out of every 30 mistakes the model makes are mistakes a person would not make.
Without the third number you have only two, and two numbers can tell you about variance but say nothing about whether the training error is any good. Teams that skip the estimate end up funding capacity work on tasks that were already at the ceiling, or accepting a training error that a fifteen-minute labelling exercise would have shown was terrible.
Lanternfish's field team hand-labelled 50 dev images cold, with no access to the model's answer. They got one wrong: a badger at the edge of the frame, in a photo where two of the three labellers also said badger and the ground truth said nothing there at all. One in 50 is 2 percent, and it came with a useful side effect: the disagreements were concentrated in exactly the night-blur category the error-analysis sheet had at the top.
Self-studyThe same three numbers on a much harder task4 min▶
Chapter 22's worked example is a system with 30 percent dev error on a task where the optimal error rate is 14 percent. The instinct in the room is always that the model is broken. The decomposition disagrees: avoidable bias is 1 point, so the model is fitting its training data to within a point of the best anything could do, and the remaining 15 points are variance.
Drawn beside Lanternfish, the two look nothing alike and are read exactly the same way. One is a task a person handles easily where the model has a generalisation problem. The other is a task that is genuinely hard where the model also has a generalisation problem. Both diagnoses point at the same project family, and neither is visible from the headline number.
Each diagnosis funds a different project
8 min liveThe decomposition is only worth an afternoon if it changes what the team does on Monday. It does, because the two removable terms have disjoint sets of remedies. Chapter 25 holds the techniques for reducing avoidable bias and chapter 27 holds the techniques for reducing variance, and picking from the wrong chapter is a quarter spent moving a term that was already at its floor.
LiveTwo families, named4 min▶
| Term | Reads as | Project family | What will not help |
|---|---|---|---|
| Unavoidable bias | Human-level error on this task | None. Change the input or the task, or accept it | Every modelling technique there is |
| Avoidable bias | Training error minus human-level | Chapter 25: make the model able to fit its training data | More training data. It does not fix data the model already sees and gets wrong |
| Variance | Dev error minus training error | Chapter 27: make what the model learned carry to unseen data | More capacity on its own, which usually widens the gap |
Regularisation, dropout and model capacity are the mechanics inside those two families, and they are taught properly in the intro ML and deep learning courses. What the decomposition contributes is the prior question: which family the next four weeks belong to. Lanternfish has one point of avoidable bias and nine of variance, so the ceiling on all of chapter 25's work is one point. That is the entire argument against the larger backbone, and it fits in a sentence.
LiveThe sheet says which category, the decomposition says which shape4 min▶
Session 3's error-analysis sheet and tonight's three numbers answer different halves of the same question, and a team needs both. The sheet says night infrared blur is 33 percent of the errors and re-labelling is worth 1.62 points per week. It cannot tell you whether those blurry frames are wrong because the model never learned them or because it learned them and cannot generalise. The decomposition says the shape of the fix is generalisation. It cannot tell you which images to point that fix at.
- Sheet first, decomposition second, on any real backlog. The sheet narrows five thousand possible projects to five. The decomposition then rules out the ones whose family does not match, and reorders what is left.
- They can disagree usefully. A category that dominates the sheet and sits in the family with almost no headroom is a project that will feel important and deliver a fraction of a point.
- They share an input. Both are computed on the dev set from session 1, and both inherit whatever is wrong with it.
- Run the decomposition per category when a category is big enough. Night blur at 42 of 120 errors can carry its own training and dev rates, and it often has a different shape from the whole.
LiveOne knob per problem3 min▶
Orthogonalization is the discipline of having a separate control for each of these terms, so that turning one does not move the others. It is course material from Ng's Structuring Machine Learning Projects rather than anything in the book, and it is worth naming plainly because it is the reason the decomposition is actionable at all.
A change that increases capacity and adds augmentation and rebalances the classes in the same pull request moves avoidable bias and variance at once. If the dev number improves, nothing has been learned about which term moved, and the next decision is a guess. One change, one term, one measurement, and the two halves from session 4 reported side by side.
Self-studyThe honest limit of all three numbers4 min▶
- Everything here rests on the dev set from session 1. If it is drawn from the wrong distribution, the variance term is measuring the distance between two populations rather than a generalisation gap, and session 7 is where that gets separated properly.
- A wrong human-level estimate moves the whole diagnosis. Estimate 5 percent when the truth is 2 and Lanternfish's avoidable bias goes negative, which reads as a model beating people and usually means the labellers had an easier job than the model does.
- Which human, on what evidence, is part of the estimate. A field ecologist with the full camera burst is a different number from a contractor with one frame. Match the labeller's information to the model's input or the estimate is about a different task.
- Fifty images gives a rough number, and rough is usually enough. One error in 50 is 2 percent with a wide band around it. It is enough to tell 2 percent from 15; it is not enough to argue about 2 against 3.
- Training error on augmented data is not training error. Compute it on the same metric and the same preprocessing as the dev number, or the subtraction is between two different things.
- The terms move as you work. Re-run the decomposition after each project rather than treating the first reading as a standing fact about the system.
Diagnose the classifier
★ 20 min · everyone buildsYou need predictions on the training set and on the dev set, the split from session 4, and fifty minutes of somebody's eyes. The Lanternfish tables are cams.predictions with a split column and cams.dev_results from last session.
Compute training error and dev error on the same metric. Same preprocessing, same threshold, same class set. Take the dev number from the Blackbox half, because the Eyeball half has been read and its rate is a diagnostic rather than a measurement.
Run a small human-labelling exercise on 50 dev images. Draw them from the Eyeball half, show the labeller exactly what the model sees and nothing more, and hide the model's answer. Two labellers if you can afford it; where they disagree is where the ground truth is soft.
Compute the three terms and check they sum. Unavoidable is the human-level rate, avoidable bias is training minus human-level, variance is dev minus training. If they do not add back to the dev error, one of the three was measured on different data.
Write the diagnosis into the results table beside the two half-errors. Same row, same run label. Three months from now, the question will be when the shape changed, and only a stored row can answer it.
Check which family the current sprint is in. Write down what is in flight, tag each item bias or variance, and compare with the diagnosis. Lanternfish had a larger backbone in flight against one point of available bias.
Re-run the decomposition per category for the big ones. Night blur is 42 of the 120 errors, enough to carry its own three numbers, and a category whose shape differs from the whole is the most useful thing this exercise produces.
A negative avoidable bias is a measurement bug, not a milestone. If training error comes out below the human-level estimate, the labellers almost certainly had information the model does not, or the two numbers were computed on different preprocessing. Fix the measurement before anybody writes the word superhuman in a deck.
Before session 6
30 min- Compute all three numbers for one model you own, on one metric, and write the decomposition down. If you cannot get a human-level estimate this week, write the number you are assuming and label it an assumption.
- Tag every item in the current sprint as bias work or variance work. Count the weeks in each column and compare with the two terms.
- Take the largest category from your error-analysis sheet and decompose it on its own. Note whether its shape matches the system as a whole.
- Find one recent change that moved more than one term at once, and write what you would have to run to work out which term actually moved.
- Bring your training and dev numbers to session 6. Learning curves are the same three numbers plotted against training set size, and they answer the question the single reading cannot: whether more data would help.
Three questions
Lanternfish reads training error 3 percent, dev error 12 percent, human-level 2 percent. What is the avoidable bias?
Avoidable bias is "calculated as the difference between the training error and the optimal error rate". Dev minus training is variance. The three terms sum back to the 12 percent dev error, which is the check.
Two teams both report 3 percent training error. Why can that be excellent work on one task and a serious failure on another?
Chapter 33's point: without a human-level estimate you cannot tell a hard problem from a weak model, and the same training error supports both readings.
A system has 30 percent dev error, an optimal error rate of 14 percent and 1 point of avoidable bias. A teammate proposes a much larger model. What does the decomposition say?
Chapter 22's worked example. A headline number that looks like a broken model is a hard task plus a generalisation problem, and capacity work has one point of headroom.