Look at 100 mistakes

Every project proposal in machine learning is an implicit claim about the errors: "the model is wrong because of X, and fixing X will help". Almost nobody checks the claim, because checking it means sitting with the wrong answers one at a time, and that feels like clerical work next to building a model. It is the opposite. It is the only step in the whole loop where a person learns what the model is actually getting wrong, rather than what they assumed it would.

The instrument is deliberately humble: a spreadsheet, a person, about a hundred misclassified examples, an afternoon. Its output is a share per suspected cause, and that share is a hard ceiling on what any project aimed at that cause can deliver. A team that reads the sheet before the sprint plan funds projects by ceiling; a team that does not funds them by who argued best.

• Dev set - the examples the team tunes and chooses on, drawn from the traffic the field will judge. Lanternfish's has 1,000 images and the model is wrong on 120.
• Misclassified example - one dev set item the model got wrong. The raw material of the afternoon.
• Error analysis - looking through misclassified examples and categorising them by suspected cause, so projects can be compared by how many errors each one touches.
• Category share - the fraction of the looked-at errors that tick a given cause. The bottom row of the sheet.
• Ceiling - the most a project can improve the error rate, which is the share of the errors it addresses. A project that fixes a 5 percent category can improve a 90 percent system to 90.5 at best.
• Points per week - a project's ceiling, in points of error, divided by the weeks it takes. The column that ranks projects.
• Eyeball dev set - the part of the dev set a person actually looks at. Lanternfish's is 500 images holding 64 of the 120 errors, which is why the shares below are shares of 64. The other half stays unseen for reasons practitioner session 4 owns.

Chapter 15 describes the mechanics in a sentence: "I usually create a spreadsheet and fill it out while looking through ~100 misclassified dev set images." One row per wrong image. One column per suspected cause, written down before starting, from whatever the team already believes is going wrong. A comments column for whatever does not fit. The person looks at the image, ticks the columns that apply, jots a phrase, moves on. When a comment repeats three times, it becomes a new column and the earlier rows get a second pass.

The bottom row is the point of the exercise: the share of the looked-at errors that tick each column. Lanternfish's afternoon, over the 64 errors in the half of the dev set the team looks at, produced night infrared blur 33 percent, deer mistaken for elk 19, animal half out of frame 19, empty frame called animal 16, and a column nobody had proposed, wrong label, 14.

About a hundred is the right size for a reason. Fewer and the shares wobble with which particular examples happened to be wrong; more and the afternoon becomes a week, and the point of the exercise is that it is cheap enough to repeat every time the model changes.

Chapter 14 makes the arithmetic explicit. If 5 percent of the misclassified examples fall in the category a project would fix, then "5% is a 'ceiling' (meaning maximum possible amount) for how much the proposed project could help". A system that is 90 percent accurate has 10 percent error; removing 5 percent of that error takes it to 9.5 percent error, so the system improves "at best" from 90 percent to 90.5. That is the outcome if the project works perfectly, and projects do not work perfectly.

The ceiling is what turns a proposal into a number. "Build a deer-versus-elk model" is an argument. "Build a deer-versus-elk model, which touches 19 percent of the errors, and if it removed 70 percent of those it would take at most 1.7 points off the error rate" is a bid, with a maximum written on the front.

Intuition about errors is intuition about the errors a person happens to remember, and people remember the interesting ones. A deer that looks exactly like an elk is a story. A frame that is too dark to see anything is not. The sheet counts both the same, which is exactly what intuition cannot do.

• The favourite category is rarely the biggest. It is usually the hardest, because hard problems are what senior people enjoy. Hard and large are different properties, and only the sheet distinguishes them.
• Causes nobody proposed show up. Wrong labels were not on Lanternfish's list. They were 14 percent of the errors and the cheapest thing on the eventual sheet to fix.
• The argument ends. Five opinions about why the model is wrong become five columns with a share under each. What remains to argue about is cost, and that is the next part.

• One person, not a committee. A single pair of eyes ticks consistently. Two people in a room negotiate every borderline case and the afternoon becomes a day.
• Someone who knows the domain. Whoever can tell a deer from an elk in infrared should be the one looking, not whoever wrote the model. At Lanternfish that was a field biologist, and she is the one who found the wrong labels.
• Start the columns from the team's beliefs and let the comments column add the rest. The sheet is allowed to disagree with the people who designed it.
• Let a row tick more than one column. A dark image with the animal half out of frame is both. Forcing one cause per row hides the overlap that decides whether two projects compete for the same errors.
• Keep the sheet. After the next model change, redo it. The columns that shrink tell you the change worked; the ones that do not tell you what it did not touch.
• Look only at the half meant for looking. Lanternfish looks at 500 of its 1,000 dev images and keeps the other 500 unseen, so that after a quarter of tuning to what it saw there is still an honest number left. Practitioner session 4 owns the why; the leader's job is to insist the unseen half exists.

The ceiling column is the share of errors times the fraction the project would actually remove, expressed in points of dev error. Wrong labels are 14 percent of the errors; re-labelling catches about 90 percent of those; at 12.8 percent error on the looked-at half that is 1.6 points. The deer-versus-elk specialist touches 19 percent, would remove perhaps 70 percent of those, and comes to 1.7 points. On ceiling alone the two projects are neighbours.

The weeks column is the estimate the team already has, because it is the same estimate they put in the sprint plan. Re-labelling is one week. The specialist model is eight. Divide, and the neighbours are 1.62 points per week against 0.21. The whole disagreement about what to do first was a disagreement about a number nobody had computed.

Sorted, the sheet reads re-label, denoise, motion filter, occlusion, specialist. The middle two are 0.84 and 0.80, near enough that the two-week job goes ahead of the three-week one so its points land earlier; the timeline below follows that order.

Both orders do the same five projects, so both finish in the same place: 3.6 percent error on the unseen half of the dev set at week 18. What differs is everything before that. Measured on the half of the dev set the team never looks at, so that the number is honest:

• Sheet order (re-label, motion filter, denoise, occlusion, specialist): 11.2 percent to 10.2 after week 1, 9.0 after week 3, 6.2 by week 6, 3.6 at week 18.
• Team order (specialist model first, then the rest): still 11.2 through week 6, 9.2 at week 8 when the specialist ships, 3.6 at week 18.

Order does not change where a team finishes. It changes how long the field lives with the error. In the team's order, the cameras ran at double-digit error for eight weeks longer than they needed to, and the product built on top of them inherited every one of those weeks.

• Every error counts the same. A rare protected species mislabelled as a common one is one tick, and a blurry deer is one tick. The sheet does not know the difference unless a severity column is added, with a weight per row, and the shares recomputed with it. A real deployment adds that column first; it reorders the middle of the sheet and does not rescue either wrong order.
• One example can tick several columns. A dark frame with the animal half out of shot is both night blur and occlusion, so the shares can sum past 100 and two projects can compete for the same errors. Fixing one shrinks the other's ceiling. Re-run the sheet after each project rather than trusting the original shares to the end.
• The shares are of the errors you looked at. 64 errors give shares that would move by a few points on a different 64. Ranking on a two-point difference in share is ranking on noise; ranking 33 against 14 is not.
• The ceiling assumes the project works. A 90 percent removal rate is a hope until it is measured. The sheet ranks bids, and a bid can still lose.
• Weeks are estimates. The same estimates the sprint plan already runs on, no better and no worse. If the team's week estimates are off by half, so is the points-per-week column, in the same proportion for every row.