The error-analysis spreadsheet

Every team has theories about why its model is wrong, and every theory has a project attached. What none of the theories have is a count. The spreadsheet is a counting device: a human looks at each misclassified example, ticks the causes that apply, and the percent row at the bottom turns opinions into shares. The whole exercise takes an afternoon for a hundred examples, which is less time than the average meeting spent arguing about which project to fund.

The counting has a second effect that is easy to miss. It surfaces categories nobody proposed. Lanternfish's team had five theories going in; wrong labels was not one of them. It turned out to be 14 percent of the errors and the best points per week on the sheet, because nobody has a theory about their own labels being wrong until they look.

• Error analysis - looking at misclassified dev-set examples by hand to understand the causes. Chapter 14 calls it looking at dev set examples to evaluate ideas.
• Category - a suspected cause of error, one per column. Night blur, deer mistaken for elk, animal half out of frame, empty frame called animal, label was wrong.
• Share - the percent of misclassified examples that tick a column. The bottom row of the sheet.
• Ceiling - the most a project can improve the error rate, if it removed every error it targets. Chapter 14's word, in quotation marks in the book.
• Removes - the share of a category's errors a project is expected to fix when it works. A denoiser will not rescue every night frame.
• Points per week - ceiling in points of dev error, divided by weeks of work. The ranking column.
• Eyeball half - the 500 dev frames you are allowed to look at. The sheet is over their 64 errors. Session 4 explains why the other 500 stay closed.

The shape is deliberately plain, and every part of it earns its place:

• One row per misclassified example. Not a sample of them, not the interesting ones. Every wrong frame in the half you are looking at.
• One column per suspected cause. Start with the team's theories. Add a column the moment you see a cause that has no column; the book's own example grew a column mid-exercise.
• Ticks. A tick where the cause applies. Nothing more precise; a cause either explains the mistake or it does not.
• A comments column. Free text for what the tick does not capture: "3am, branch in wind". The comments are where the next column comes from.
• A percent-of-total row at the bottom. The share of rows that tick each column. This is the row the whole exercise exists to produce.

A spreadsheet, not a labelling tool, because the person filling it in has to be the person who will decide what to build. Delegating the looking to an annotation vendor produces a sheet nobody on the team has read.

Two rules that trip people up. First, an image can tick more than one column: a night frame with an animal half out of shot is both night blur and occlusion, and both projects would help with it. So the percent row does not have to add to 100, and on the Lanternfish sheet it adds to 101 because of that overlap plus rounding. Each column's share is still a correct statement about how many errors that project could touch.

Second, the sheet is over the Eyeball half only: 64 errors, not 120. The Blackbox half stays unopened, and session 4 explains what it is for. The consequence tonight is that the shares are estimates from 64 examples, and the current error rate the ceilings are computed against is the Eyeball half's 12.8 percent, since that is the half the sheet describes.

Chapter 14 works the arithmetic on a cat detector where 5 percent of the misclassified images are dogs: "5% is a 'ceiling' (meaning maximum possible amount) for how much the proposed project could help", so a 90 percent accurate system improves at best to 90.5 percent. The reasoning is two steps: a project that targets one category can remove at most that category's share of the errors, and the errors are a fraction of the dev set, so the improvement in accuracy is the share times the error rate.

The book's ceiling assumes the project removes every error in its category. A realistic sheet adds one more factor, the share the project is expected to remove when it works, because a denoiser does not rescue every night frame and a motion filter does not catch every empty one. The three-factor version is what Part 2 uses:

ceiling in points = share of errors x share the project removes x current error rate

For wrong labels on Lanternfish: 14 percent of the 64 errors, a re-labelling pass that fixes 90 percent of them, an Eyeball error rate of 12.8 percent. 0.14 x 0.9 x 12.8 is 1.6 points. That is the most the project can do; it is not a forecast of what it will.

The weeks column is an estimate the team already makes for every project, usually in a planning meeting far from the spreadsheet. Bringing it onto the sheet is the whole trick. Once it is beside the ceiling, dividing one by the other is a formula in a cell, and the sort order that comes out is the order of work.

On the Lanternfish sheet the ranking reads: re-label the dev set at 1.62 points per week, the infrared denoising pass at 0.84, the motion-trigger filter at 0.80, occlusion augmentation at 0.30, and the deer-versus-elk specialist model at 0.21. The denoiser and the motion filter are within 0.04 of each other, which is inside the noise of a 64-example estimate; either order for those two is defensible. The top and the bottom of the list are not close, and that is where the decision lives.

Wrong labels are the smallest category on the sheet at 9 of 64, and the best points per week at 1.62, because re-labelling is a one-week job that removes 90 percent of what it targets. This pattern repeats across projects and domains: label noise is never anyone's theory, so it never has a project until the sheet gives it one, and when it does the project is cheap. It also has a second payoff the sheet does not show: every other project's ceiling was computed against errors that include the mislabelled ones, and once they are gone the remaining shares are cleaner.

• Every error counts the same. A mislabelled endangered species and a blurry deer are each one row. A real deployment weights them, and a severity column is the first extension worth adding. It reorders the middle of the sheet; it does not usually rescue the eight-week project.
• The shares are estimates from 64 examples. A category at 14 percent is really "somewhere around 14": nine frames, and one more or fewer would move it a point and a half. Treat differences of a few points between columns as ties.
• Removes is a guess until the project ships. The 60 percent for the denoiser is an engineer's estimate. Write it down so it can be checked against what the project actually did, and adjust the next sheet.
• Weeks are estimates too, and they are usually optimistic in the same direction for every project, which is why the ranking survives it: an eight-week project that takes twelve is still last.
• The sheet describes the Eyeball half. The 12.8 percent the ceilings are computed against is that half's rate; the Blackbox half sits at 11.2 and will be the number reported when the projects ship.

None of these limits argue against the sheet. They argue against reading 0.84 versus 0.80 as a decision, while still reading 1.62 versus 0.21 as one.