Why the metric is a leadership decision
Session 1 made the case that a machine learning project is a loop, and that the speed of the loop is what the funder actually controls. Every turn of that loop ends with the same act: somebody reads the dev set and decides whether the idea stays. The metric is what they read. If it takes two numbers to say whether the model got better, every turn of the loop ends in a meeting, and the meeting is the loop speed. Lanternfish's team lost three weeks of a quarter to exactly that meeting, before anybody had written down what the field cared about.
★ What you walk out with tonight. The reason one number beats two: not because one is more accurate, but because one is a decision and two is a debate. The book's pattern for the other requirements, one metric to optimise and the rest as bars to clear, so that "fast enough" and "small enough" stop competing with "accurate". A one-sentence metric statement for a model you fund, with what it hides written next to it. And the signal that says it is time to rewrite the sentence.
LiveWhat the metric decides, and who should decide it3 min▶
The metric is the definition of "better". Every model change the team accepts is accepted because it moved the metric, and every change it drops is dropped because it did not. So the metric silently decides what the model becomes over a quarter: which mistakes it learns to avoid, and which it is allowed to keep making because they cost nothing on the number.
That is a product decision wearing a technical costume. Which mistakes matter is a question about the field, the customer and the mission, and an engineer asked to choose a metric alone is being asked to make that call without the information. The right shape is a sentence the funder can read and sign: what the field cares about, the one number that tracks it, what that number hides, and the bars everything else has to clear.
Lanternfish's classifier was scored on accuracy across the species set because accuracy is what the training library printed. Nobody chose it. Two quarters later the conservation groups asked why the one species they were legally required to report was the one the model most often missed. Accuracy had never been asked to care.
Self-studyNames, not definitions3 min▶
The words below appear tonight only as names. Each has a definition, a threshold behaviour and a failure mode, and the model-evaluation course owns all three. Here the only question is which one goes in the sentence.
- Accuracy - the single number Lanternfish currently optimises, over the whole species set.
- Precision and recall - the pair most classification teams are shown, and the pair most often argued about.
- F1 and its relatives - single numbers built by combining a pair, so that a team has one thing to optimise.
- Per-species accuracy - the same number computed for each species separately, which is what makes a rare species visible.
- Inference time, model size, battery draw - the non-accuracy requirements a camera in a forest imposes, and the ones that become bars rather than targets.
Establish a single-number evaluation metric for your team to optimize
9 min liveThe chapter title is the instruction. A team needs one number that goes up when the model gets better, so that at the end of every turn of the loop the decision is a comparison rather than a conversation. The cost is that any single number hides a trade inside it. The discipline is to choose it knowing what it hides, rather than letting the training library choose for you.
LiveTwo numbers is a debate, one number is a decision3 min▶
Put two models in front of a team with two numbers each, and one model will usually win on one number and lose on the other. The team now has to decide how much of one it would trade for how much of the other, and that is a values question, so everybody has a view and nobody has the authority. The meeting runs long, the decision is deferred, and the same meeting happens after the next experiment because nothing was settled.
Put the same two models in front of the same team with one number each, and the higher one wins. The decision takes the time it takes to read two numbers. The trade has not gone away; it was made once, when the single number was chosen, by whoever chose it. That is the whole argument for a single-number metric: it moves the values question out of every sprint and into one deliberate decision.
- The loop turns faster because each turn ends in a comparison, not a conversation.
- The team stops relitigating because the trade was settled above them, in writing.
- Progress becomes visible across weeks: one line going up or not, rather than two lines crossing.
LiveWhat a single number costs3 min▶
Any single number is a compression. It takes every kind of mistake the model can make and adds them up according to some rule, and the rule decides which mistakes are expensive and which are free. Accuracy's rule is that every wrong frame costs the same, so a mistake on a species that appears in half the frames and a mistake on one that appears in one frame in a hundred are worth exactly the same to the number. That is not wrong; it is a choice, and it was made the moment accuracy was picked.
So the cost of a single number is that it hides the trade inside it, and the discipline is to choose it knowing what it hides. Written next to the metric, in the same sentence, should be the one thing the number is blind to that the field cares about. If nobody can write that clause, nobody has thought about the metric yet.
LiveLanternfish chose accuracy, and what that hides3 min▶
Lanternfish optimises accuracy over the species set: the share of dev frames where the predicted species is the labelled one. It is a defensible single number for a classifier whose customers want to know what walked past. It is also blind to one thing the customers care about more than anything else.
One of the species on the list is protected, and it appears in about 1 percent of frames. A model that never recognises it at all loses at most one point of accuracy. A model that gets it right every time gains at most one point. To the number, the protected species and a common deer are interchangeable; to the conservation groups paying for the cameras, the protected species is the reason the cameras exist.
Two quarters of accuracy-driven work made the classifier steadily better on deer, elk and boar, and steadily no better on the one species with a legal reporting requirement. Nothing in the loop could have caught it, because nothing in the loop was looking. The number went up every sprint, honestly.
Self-studyThree ways teams end up with the wrong single number4 min▶
- The library chose it. Accuracy is what the training tool prints by default, so it becomes the metric without anybody deciding. The fix is a sentence with a signature on it.
- The convenient one was chosen. The metric that was easy to compute on the data to hand, rather than the one that tracks what the field cares about. It measures the wrong thing precisely, exactly as a dev set from the wrong traffic does.
- The metric was chosen for the dashboard. A number that looks good to report is not the same as a number that is hard to move in the wrong direction. Numbers chosen to look good tend to look good.
All three share a cause: the choice was made by whoever was nearest the keyboard, on the day the project started, and never revisited. The build-along makes the choice explicit and gives it an owner.
Optimizing and satisficing metrics
8 min liveA single number to optimise does not mean nothing else matters. A classifier on a camera in a forest has to run fast enough, fit in the memory the camera has and not drain the battery, and none of those is accuracy. Chapter 9 gives the pattern that keeps them from turning back into a debate: one metric you optimise, and the rest as bars the model only has to clear.
LiveOne to optimise, N-1 to clear3 min▶
Chapter 9 is titled "Optimizing and satisficing metrics", and the pattern is exactly that. The book's example is a classifier where accuracy is the optimizing metric, the thing you want as high as possible, and running time "should take at most 100ms" is a satisficing one: it has to clear the bar, and once it does, faster earns nothing. With N things you care about, you set N-1 of them as satisficing bars and optimise the one that is left.
The reason this works is that a bar cannot start an argument. Either the model runs in under the limit or it does not; there is no trade to negotiate, so there is no meeting. All of the negotiating happens once, when the bar is set, and setting it is a question the funder can answer: how slow is too slow for the field, how big is too big for the device.
- A satisficing metric is pass or fail. A model that fails it is out, however accurate. A model that passes it is compared on the optimizing metric alone.
- Every bar has a number. "Fast enough" is an argument; "at most 200 milliseconds" is a gate.
- There is exactly one optimizing metric. Two things to maximise is the debate from Part 1 wearing a different hat.
LiveLanternfish's stack2 min▶
The camera is a small computer on a tree, running on a battery, with a fixed amount of flash storage and a motion trigger that fires many times a night. Those facts set the bars. Accuracy is the one number that goes up; everything else is a gate with a threshold the hardware team can defend.
| Kind | Metric | Rule | Set by |
|---|---|---|---|
| Optimizing | Accuracy on the species dev set | As high as possible; hides the 1 percent protected species | Product, with the field |
| Satisficing | Inference time on the camera | At most 200 milliseconds per frame | Hardware, from the trigger rate |
| Satisficing | Model size | At most 25 megabytes | Hardware, from the flash budget |
| Satisficing | Any further requirement | A bar with a number, never a second optimizing metric | Whoever owns the constraint |
Once this table exists, the eight-week specialist model from session 1 has to clear two gates before its accuracy is even read. A model that is two points more accurate and 40 megabytes is not a trade-off to discuss. It is out.
LiveWhen to change the metric3 min▶
Chapter 11 covers when to change dev sets and metrics, and the signal for the metric is one sentence: change it when it has stopped measuring what the field cares about. The metric is a proxy for a goal outside the model, and a proxy can drift away from its goal without anybody doing anything wrong. When the number and the field disagree about which of two models is better, the field is right and the number is what changes.
Lanternfish's signal was the protected species. Accuracy ranked a model that never recognised it above one that sometimes did, because the second model made a few more mistakes on deer. The field would have taken the second model without hesitation. That disagreement is the whole test: the metric had a preference the customer did not share, so the metric was wrong, not the customer.
- The fix is a new sentence, not a quiet tweak. Per-species accuracy with the protected species weighted, or a satisficing bar on the protected species alone, written down and signed again.
- The dev set usually changes with it, because a species that appears in 1 percent of frames needs enough examples in the dev set to be measured at all. Practitioner session 1 owns that arithmetic.
- What is not a signal: a number the team dislikes. A metric that says the model is worse than hoped is doing its job.
Self-studyThe honest limit of a single number3 min▶
A single-number metric does not make the trade go away. The protected species is still worth one point to accuracy after the sentence is written, and the deer-versus-boar confusion is still worth more. What changes is where the trade is made and by whom. Before the sentence, it was made implicitly, every sprint, by whichever model happened to score higher. After the sentence, it was made once, deliberately, by the person who is accountable for the field, and written down where the team can read it.
- The trade is visible. The "what it hides" clause names it, so nobody is surprised when the field raises it.
- The trade has an owner. Changing it is a decision with a signature, not a drift.
- The trade is revisited on a signal, not on a schedule and not on a mood.
That is the modest claim for a single number: it turns a weekly argument into a deliberate choice. It does not turn a hard trade into an easy one.
Write the metric sentence
★ 18 min · everyone buildsBring a model you fund. If you have none to hand, use Lanternfish. The output is five lines of plain prose that the person who signs off the model could read in a minute and object to in two.
Write the one sentence that says what the field cares about. Not what the model does; what the customer would be angry about. Lanternfish: "Conservation groups need every frame identified to species, and a protected species must never go unreported."
Choose the single optimizing metric that best tracks it, and write what it hides. Lanternfish: accuracy on the species dev set, which hides the protected species because it is 1 percent of frames and worth one point at most. If the hidden thing is the thing the field cares about most, the metric is wrong already, and the next step is to change it before the sentence is signed.
List every other requirement as a satisficing bar with a number. Inference time at most 200 milliseconds on the camera. Model size at most 25 megabytes. If a requirement resists having a number, it is not yet a requirement; it is a wish.
Write the signal that would make you change the metric. Lanternfish: "If the metric ranks a model that misses the protected species above one that catches it, the metric changes, not the model." A signal written in advance is a decision; one discovered in a complaint is a crisis.
Read the sentence to the person who signs off the model. Out loud, in a meeting, and watch which clause they push back on. It is almost always the hidden-thing clause, and that push-back is the metric being chosen properly for the first time.
The bad version is not wrong, it is undecided. Every word in it is true and none of it can be checked. "Accurate and fast" is two things to optimise, "good precision and recall" is the three-week meeting, and "small enough" has no number. The good version can be failed, which is the only way a sentence can be useful.
Before session 3
30 min- For one model you fund, find out what number the team actually optimises and who chose it. If the answer is "the library" or "nobody", write the metric sentence and take it to them.
- Write the "what it hides" clause for that number. If you cannot, ask the team what mistake the model could make every day without the number moving. That is the clause.
- List the requirements the team currently treats as things to improve, and turn each into a bar with a number. Count how many meetings the last quarter spent on the ones that should have been bars.
- Ask the field, a customer or a downstream team, which of the last two model versions they preferred. If their answer disagrees with the metric's, the metric changes, and session 3 will show you where the errors that matter actually are.
Three questions
A team is shown two models: one has higher precision, the other higher recall. Why does the book insist on a single-number metric here?
Two numbers is a debate, one number is a decision. The trade does not disappear; it is made once, by whoever chooses the metric, instead of every sprint.
Lanternfish requires inference under 200 milliseconds on the camera. A new model runs in 150 milliseconds. What does that earn it?
Chapter 9: the running-time requirement is satisficing. Clear it, then stop caring. Faster than the bar earns nothing, which is exactly why a bar cannot start an argument.
Accuracy ranks a model that never recognises the protected species above one that sometimes does, because the second makes a few more mistakes on deer. The field would take the second model without hesitation. What changes?
Chapter 11: change the metric when it disagrees with the field about which model is better. The metric is a proxy; the field is the goal.