Where the error comes from

"The model is 12 percent wrong" is a fact, and it is not a decision. Twelve percent because the model is too simple to fit the pattern is fixed one way; twelve percent because the model fits the training data beautifully and falls apart on anything new is fixed the opposite way; and some of the twelve is not the model's fault at all, because the frames are too dark for anyone to read. Three different problems, one number, and every proposal in the backlog silently assumes one of the three.

The split takes two subtractions and no new experiments. A team that does the arithmetic funds the source that is actually large. A team that skips it funds whichever source the loudest proposal assumed, and finds out in three months.

• Training error - how often the model is wrong on the data it learned from. Lanternfish: 3 percent.
• Dev error - how often it is wrong on the dev set, data it did not learn from, drawn from the traffic the field will judge. Lanternfish: 12 percent.
• Human-level error - how often a competent person is wrong on the same task with the same inputs. Lanternfish's field biologists: 2 percent on the same frames.
• Optimal error rate - the error no model could get below, because some inputs do not carry the answer. Also called unavoidable bias. In practice estimated by human-level error.
• Avoidable bias - training error minus the optimal error rate. What the model fails to learn even from the data it has.
• Variance - dev error minus training error. What the model fails to carry over from the data it saw to the data it did not.
• Bias, loosely - the training error as a whole, unavoidable and avoidable together. Only the avoidable part is a target.

Chapter 22 works the arithmetic on a system with 30 percent dev error. Some of that error is structural: the inputs are so noisy that the best possible system would still be wrong 14 percent of the time. That 14 is the optimal error rate, also called "unavoidable bias", and no project on any backlog can touch it. The next slice is avoidable bias, 1 percent, "calculated as the difference between the training error and the optimal error rate": the model is wrong on its own training data 15 percent of the time, and 14 of that was unavoidable, so 1 point is the model's own failure to learn. The rest is variance, dev error minus training error, which is 30 minus 15: everything the model learned on its training data and failed to carry over.

The same three subtractions on Lanternfish: training error 3, dev error 12, human-level 2. Unavoidable bias 2. Avoidable bias 3 minus 2, which is 1. Variance 12 minus 3, which is 9. Nine points of variance against one of avoidable bias. The diagnosis is variance, and it took thirty seconds once the three numbers were on the table.

The optimal error rate cannot be measured directly; nobody has the perfect model to read it off. Chapter 33 gives the practical substitute: on tasks people do well, human-level performance is the best available estimate of it. If field biologists are wrong 2 percent of the time on the same infrared frames, then roughly 2 percent of those frames do not carry enough information to answer, and a model that reaches 2 percent has reached the floor.

Human-level error does two jobs at once. It tells the team how much of the training error is avoidable, which is the first subtraction. And through that, it tells them which of the two sources is worth attacking: a model at 3 percent training error against a 2 percent human is nearly done learning from its data, so the 9 points of dev error above that cannot be a learning problem. Without the human number, 3 percent training error looks like room to improve, and a team will spend the quarter improving it.

• Same inputs the model sees. A biologist with the full video clip and a map of the camera site does better than one with the single frame the model gets. The yardstick has to be measured on the model's inputs, or it will read too low and make avoidable bias look larger than it is.
• Competent people, not average ones. The optimal error rate is a floor, so the estimate should come from whoever does the task best. A team of specialists agreeing beats one person guessing.
• A sample of the dev set, not a different set. Human-level error on daytime web photos says nothing about night infrared frames. Fifty to a hundred dev frames, scored by hand, is usually enough to place the number within a point.
• Write down the date. A human-level estimate from a grant application two years ago was measured on different cameras. It is a number, and it is not the yardstick.
• When people are bad at the task, a stock forecast, a click prediction, human-level error stops being a good proxy for the optimal error rate and the decomposition gets harder to read. Say so rather than pretending the number means what it means for species in a photo.

Chapter 25 lists the projects that reduce avoidable bias, and they are all ways of letting the model learn more from the data it already has: a bigger model, training for longer, better features. Chapter 27 lists the projects that reduce variance, and they are ways of stopping the model from learning the training data too specifically: more training data, regularisation, a simpler model. The two lists are close to opposites, and that is the point. A bigger model helps bias and can hurt variance; more data helps variance and does nothing for bias.

• Bias diagnosis: the model is wrong even on what it trained on, by more than a person would be. Fund the chapter 25 family. Ceiling: the avoidable bias. For Lanternfish, 1 point.
• Variance diagnosis: the model is nearly as good as a person on its training data and much worse on new data. Fund the chapter 27 family. Ceiling: the variance. For Lanternfish, 9 points.

The names are enough for the decision. Whether regularisation is done one way or another, what a "bigger model" means for a given architecture, is technique, and the intro machine learning and deep learning courses own it. The leader's question is only whether the sprint is drawing from the family the numbers point at.

Andrew Ng's Structuring Machine Learning Projects course, week 1, gives the design principle behind the two lists a name: orthogonalization. This is course material, not a chapter of the book. The idea is that a well-run project has one knob per problem: a set of changes that move training error and leave variance alone, and a different set that move variance and leave training error alone. Then a change can be judged by the one number it was meant to move.

The practical consequence for a leader is a suspicion. A proposed change that moves two of the three numbers at once, training error down and dev error up, say, is a change nobody can reason about, because there is no way to say which effect was intended and whether the trade was worth it. Ask for changes that pull one knob, measure the one number, and keep or discard on that.

• Dev error is only as good as the dev set. If the dev set is drawn from the wrong traffic, the variance number is the distance to the wrong place. Practitioner session 1 builds the set; a leader's check is one question: if the model were perfect on this dev set, would the field be satisfied?
• Human-level error is only as good as the measurement. Measured on different inputs, on an easier sample, or by average rather than competent people, it shifts the boundary between unavoidable and avoidable bias and can turn a variance diagnosis into a bias one.
• The decomposition says which source, not which cause. Nine points of variance does not say whether the fix is more night frames or fewer parameters. That is why session 3's spreadsheet comes first: the sheet says what the errors are, the three numbers say whether more learning or more data would remove them.
• A model can have both problems. Avoidable bias 4 and variance 5 is two doors to walk through, in order of size. The numbers still decide the order; they just do not let the team stop after one.
• The numbers move after every project. Recompute all three when the model changes. A variance project that worked will have shrunk the gap, and the next diagnosis may be different.