What the bench measures
Two error rates and a week count. The Blackbox half is the number you would report; the Eyeball half is the number you have been staring at; the week count is what the order cost. Any single one of the three can be made to look good on its own, which is why the bench refuses to show fewer than three.
★ What you walk out with tonight. An order of work you have scored rather than argued for. The measured cost of funding the interesting project first, which is eight weeks at 11.2 percent error to reach the same place. And the two-halves picture of what happens when a team tunes to the examples it looked at: one half drops to 3.8 percent, the other stays at 11.2, and the gap is not a model.
LiveCeiling is the only honest way to rank a project3 min▶
A project's ceiling is the share of current errors it could remove, times the share it actually does remove when it works. A category holding 14 percent of the errors, fixed by a project that catches 90 percent of them, has a ceiling of about 12.6 percent of your error rate. At 12.8 percent dev error that is 1.6 points, and it takes one week. A category holding 19 percent of the errors, fixed by a project that catches 70 percent, is 1.7 points and takes eight weeks. The ceilings are close; the points per week are 1.62 against 0.21.
Nobody funds the eight-week project because they computed that. They fund it because it is interesting, and a spreadsheet with a points-per-week column is what makes "interesting" visible as a cost.
Lanternfish's most senior engineer wanted the deer-versus-elk model. It was a genuinely hard problem and a genuinely good piece of work. It also shipped in week 8 to a dev set that was still 11.2 percent wrong, when a one-week re-labelling job and a two-week motion filter would have had the team at 9.0 percent by week 3.
Self-studyWhat is real here, and the one construction3 min▶
- The dev set. 1,000 images from a fixed seed, 120 wrong, each carrying the category a human wrote in the spreadsheet: night blur 42, deer-versus-elk 24, half out of frame 22, empty frame 18, wrong label 14. Split by index into Eyeball and Blackbox halves of 500; random placement left 64 errors in one and 56 in the other, which is why they start at 12.8 and 11.2.
- The projects. Each removes a stated share of its category's errors, earliest-indexed first, so the arithmetic is deterministic and the same for every learner.
- The construction. "Hand rules against the errors you looked at" removes 70 percent of errors in the Eyeball half and none in the Blackbox half. That is not modelled human behaviour; it is what a rule written against sixty specific examples does by definition, and the widget says so.
- Nothing else is modelled. No estimate of anybody's behaviour anywhere. Every number is a count.
The spreadsheet, and the numbers it actually produces
10 min liveEvery number here is computed by the bench on this page. The first table is the error-analysis spreadsheet with the two columns most teams leave off. The second is what three orders of work do to the Blackbox half over eighteen weeks.
| Category | Eyeball errors | Share | Project | Weeks | Ceiling | Points per week |
|---|---|---|---|---|---|---|
| Night infrared blur | 21 | 33 percent | Infrared denoising pass | 3 | 2.5 | 0.84 |
| Deer mistaken for elk | 12 | 19 percent | Deer-vs-elk specialist model | 8 | 1.7 | 0.21 |
| Animal half out of frame | 12 | 19 percent | Occlusion augmentation | 4 | 1.2 | 0.30 |
| Empty frame called animal | 10 | 16 percent | Motion-trigger filter | 2 | 1.6 | 0.80 |
| Label was wrong | 9 | 14 percent | Re-label the dev set | 1 | 1.6 | 1.62 |
| Order of work | Blackbox error at week 6 | At week 8 | At week 18 | Eyeball error at the end |
|---|---|---|---|---|
| Best points per week: re-label, motion filter, denoise, occlusion, specialist | 6.2 percent | 6.2 | 3.6 | 4.4 |
| Most interesting first: specialist model, then the rest | 11.2 percent | 9.2 | 3.6 | 4.4 |
| Tune to what you looked at: hand rules, 2 weeks | 11.2 percent | - | - | 3.8 percent |
Same destination, different journey. Both honest orders end at 3.6 percent after eighteen weeks, because they do the same five projects. The difference is that one team was at 6.2 percent by week 6 and the other was still at 11.2, waiting for the interesting thing to ship. Order does not change where you finish; it changes how long the field lives with the error.
Choose the order yourself
★ 25 min · everyone buildsThe bench is live below. The sheet at the top is already sorted by points per week. Press an order, or fund projects one at a time, and watch both halves.
Read the sheet before touching anything. The biggest category is night blur at 33 percent. The best points per week is the smallest category, wrong labels at 14 percent, because it costs one week. Ceiling and cost are two columns, and the ranking needs both.
Press "best points per week". Blackbox error 11.2 to 10.2 after one week, 9.0 after three, 6.2 after six. Eighteen weeks in, 3.6 percent. Notice the Eyeball and Blackbox lines move together the whole way: every fix was a fix to the problem, not to the examples.
Press "most interesting first". The specialist model ships in week 8. Until then the Blackbox line is flat at 11.2. The same five projects then follow, and the line lands at 3.6 percent in week 18, exactly where the other order landed. Hover the week-8 row and read the note.
Press "tune to what you looked at". Two weeks. Eyeball 12.8 to 3.8; Blackbox 11.2 to 11.2. This is the dev set you have memorised, and a team that only tracked the number it was looking at would ship it and call it a 70 percent error reduction.
Clear it and build your own order, one project at a time. Try the denoising pass first: the biggest category, a sensible instinct, 0.84 points per week. Then check what the re-label job would have done in the week you spent on setup.
Write down the order you would defend to the team, and the Blackbox number you would report at week 6. If the interesting project is anywhere in the first three slots, say out loud what the field is getting for the wait.
What the bench leaves out, deliberately. Every error counts the same. A mislabelled endangered species is not a blurry deer, and a real spreadsheet would weight them. Add a severity column first when you build your own; it reorders the middle of the sheet and does not rescue either wrong order.
Closing the track
45 min- Run your own error-analysis spreadsheet from session 3 through the two extra columns: weeks per project and points per week. Sort by the last column and compare with the order the team is actually running.
- Report your Eyeball and Blackbox numbers side by side for the last three model changes. If the Eyeball number moved and the Blackbox did not, you know which kind of change it was.
- Find the project in your backlog with the highest ceiling and the lowest points per week. Write the sentence that explains to its sponsor what the field gets for the wait.
- Add a severity weight to your own spreadsheet and see which rows change order. Then take the one-page memo from leader session 6 to whoever approves the sprint.
Three questions
Re-labelling the dev set fixes the smallest category on the sheet, yet ranks first. Why?
Ceiling and cost are two columns. Ranking on ceiling alone puts the specialist model near the top; ranking on points per week puts it last.
Both honest orders end at 3.6 percent Blackbox error after eighteen weeks. What did the order change?
The same five projects reach the same place. The order decides whether the team spends eight weeks at double-digit error waiting for the interesting project to ship.
Hand rules against the sixty Eyeball errors take the Eyeball half from 12.8 to 3.8 percent in two weeks and leave the Blackbox half at 11.2. What has the team built?
This is why the Blackbox half exists. A rule written against sixty specific images fixes those sixty images. The half you never looked at is the only honest reading left.