Strategy is deciding what not to try

The engineers on a machine learning team decide how to build. The person who funds the team decides what gets built, in what order, and how long each thing is allowed to take before somebody asks whether it is working. Those three decisions set the speed of the loop, and the speed of the loop sets how much the team learns per quarter. A leader who approves one large project for the quarter has, without meaning to, decided that the team will learn once.

The uncomfortable part is that the slow version usually looks like rigour. A clean architecture, a proper data pipeline, a launch when it is ready. The fast version looks scrappy: a baseline in a week, a dev set that is good enough, six things tried and five dropped. Over a quarter the scrappy team has six readings of reality and the rigorous team has one, and the six readings win almost every time.

• Dev set - the sample of real traffic that every model change is measured on. If it is drawn from the wrong traffic, every decision is about the wrong problem. Practitioner session 1 builds one.
• Dev error - the share of the dev set the model gets wrong. Lanternfish: 120 of 1,000, 12 percent.
• The loop - idea, code, experiment, read the dev set, decide. One turn of the loop is one thing learned.
• Baseline - the first system, built quickly, whose only job is to make the loop start turning. Chapter 13.
• Ceiling - the most a project could possibly improve the error rate, which is the share of current errors in the category it fixes. Chapter 14. Session 3 computes it from a spreadsheet.
• Points per week - ceiling divided by cost in weeks. The one column that ranks projects honestly.
• Eyeball and Blackbox halves - the half of the dev set the team looks at, and the half it never does, so the second half can still tell the truth. Session 3 explains why; the numbers tonight come from the Blackbox half.

Two teams start the same problem on the same day. One picks a better architecture and builds it carefully over the quarter. The other picks something ordinary, has it scoring on a dev set in a week, and then tries an idea every fortnight. At the end of the quarter the second team has read the dev set six times, has dropped five ideas that sounded good and did not work, and has kept one that did. The first team has read it once, on launch day.

The second team is ahead, and not because its people are better. It knows six things about the problem that the first team does not, and the sixth thing it learned was only visible because of the first five. Ideas in machine learning compound through readings, and readings only happen when the loop closes.

The consequence for whoever funds the work: the question to ask about a project is not "is this the right idea" but "how many times will we find out". A quarter that answers "once" is a bet, not a plan.

Chapter 13 of Machine Learning Yearning has a title that is also the whole instruction: "Build your first system quickly, then iterate." The first system is not the product. Its only job is to make the loop start turning, so that the team's next idea is chosen from what the dev set says rather than from what the team believed before it had any evidence.

Most teams violate this rule for a reason that sounds responsible: they want to do it properly. Properly means a clean pipeline, the right architecture, labels reviewed, a launch when it is ready. Each of those is a good thing. Together they push the first reading of reality to the end of the quarter, which is the one place it cannot help anybody.

• A first system in a week is a deliberate choice of an ordinary model on the data to hand, scored on a dev set that reflects the field. Practitioner session 2 does it.
• The first number will be disappointing. That is the point. A disappointing number in week one is the cheapest information the project will ever buy.
• Everything after that is iteration, and iteration is what the rest of the track is about: which of the many possible next steps to fund.

Lanternfish sells trail cameras to conservation groups, and the cameras are supposed to say which species walked past. The team spent six months on a species classifier with the architecture everybody agreed was right. Throughout, the dev set said 94 percent. The dev set was 400 photos scraped from the web: daytime, sharp, animal in the centre of the frame. The cameras shoot night infrared with animals half out of shot. In the field the classifier scored 78.

The fix was not a better model. The team rebuilt the dev set from 1,000 real camera frames, scored the same model, and got 88 percent: 120 wrong. Six months of work had produced a number that was true, and the number was 12 percent error. That is where the project started. Every session after this one is about what to do with those 120 mistakes.

• No dev set from the field. Without one, an experiment cannot be read, so the loop never closes. Owner: the team, in week one, with the funder asking for it by name.
• A metric the team argues about. Two numbers is a debate every sprint. Owner: the funder, session 2.
• Projects sized by the quarter. A project that takes twelve weeks turns the loop once. Owner: the funder, who approves the size.
• Approval between turns. If each idea needs a meeting, the meeting cadence is the loop speed. Owner: the funder, who can pre-approve the quarter's list and ask only for the readings.

Three of the four are the funder's. Engineers can shorten a training run; they cannot shorten a governance cycle they do not own.

Every machine learning team has roughly five projects it could do next, and every one of them is plausible. That is what makes the decision hard: none of the five is stupid. The denoising pass attacks the biggest category of errors. The specialist model attacks the hardest and most interesting one. Re-labelling the dev set is admitting the labels were wrong, which nobody enjoys. The motion filter is somebody else's team. The augmentation is what the last conference talk recommended.

In the absence of a sheet, the one that gets funded is the one with the most convincing sponsor, and the most convincing sponsor is usually the most senior engineer arguing for the most interesting problem. That is not a character flaw; it is what happens when the only inputs are opinions. The remedy is to give the decision two numbers it did not have.

The first number is the ceiling. Chapter 14 puts it plainly: if 5 percent of the errors fall in the category a project fixes, then "5% is a 'ceiling' (meaning maximum possible amount) for how much the proposed project could help", and a 90 percent accurate system improves "at best" to 90.5 percent. A project cannot fix errors that are not in its category, however well it is built. Session 3 shows how the share comes out of a spreadsheet of 100 mistakes; tonight the shares are given.

The second number is the cost in weeks. Engineers can estimate this to within a factor of two, which is all the ranking needs. Divide ceiling by weeks and the five projects fall into an order that has nothing to do with who is sponsoring them.

• Re-labelling the dev set fixes the smallest category, 14 percent of errors, but costs one week: 1.6 points of ceiling, 1.62 points per week.
• The deer-versus-elk specialist model fixes 19 percent of errors at 70 percent effectiveness, 1.7 points of ceiling, and costs eight weeks: 0.21 points per week.
• The two ceilings are close. The costs differ eightfold. Nobody who computed both numbers would fund the specialist first; everybody who computed only the first would.

Order does not change where the team finishes. The same five projects remove the same errors, so both honest orders land at 3.6 percent Blackbox error after eighteen weeks. What order changes is how long the field lives with the mistakes on the way there.

• Points-per-week order (re-label, motion filter, denoise, occlusion, specialist): Blackbox error 11.2 at the start, 6.2 percent by week 6, 3.6 at week 18.
• Most interesting first (the specialist model, then the rest): still 11.2 percent at week 6, 9.2 at week 8 when the specialist finally ships, 3.6 at week 18.
• The difference is eight weeks during which every camera in the field was wrong on more than one frame in ten, for a project that ended up in exactly the same place.

Choosing one project is easy; every team does it. Strategy is being able to say which four of the five you will not do this quarter, and being able to show why with a sheet rather than a hunch. The sheet matters because the sponsors of the four are still in the room, and "I have a feeling" does not survive the meeting. "Yours is 0.21 points per week and this one is 1.62; yours is fourth in the queue and starts in week 10" does.

• The sheet is the decision, not the justification. Compute it first, then decide. A sheet built after the choice is a press release.
• A no with a date is not a no. The specialist model is not cancelled; it is fifth, and week 10 is written next to it. Sponsors accept a queue far more readily than a rejection.
• The sheet will be wrong in the details and right in the order. Estimates of weeks are rough; the gap between 1.62 and 0.21 survives any estimate being off by half.
• Every error counts the same on this sheet. A mislabelled protected species is not a blurry deer, and a real sheet would add a severity column. It reorders the middle and does not rescue the interesting project.