Precision, recall, and money

When a team reports "precision 21.5%, recall 63.6%," they are reporting one point on a curve - the point their threshold happens to sit on. Move the threshold and both numbers move, in opposite directions. So the honest questions are never "what is the precision?" but:

• What does the whole curve look like? The curve is the model's actual capability; any single P/R pair is one chosen point on it.
• Which point are we operating at, and who chose it? If the answer is "0.5, the default," then the trade between finance's pain and marketing's pain was made by a library setting.
• What would it take to move? Nothing - no retraining, one number. The threshold is the cheapest model improvement that exists, which is why leaving it at the default is so expensive.

scikit-learn is blunt about this in its threshold-tuning guide: the 0.5 cutoff is "most certainly not ideal for most use cases," and the library now ships a tool that tunes the threshold against a business scorer instead. The toolmakers have already conceded the point; most org charts have not caught up.

So where SHOULD the dial sit? A 2001 paper by Charles Elkan, "The Foundations of Cost-Sensitive Learning," settled it, and the leader version is one sentence: the right threshold follows from the cost ratio of the two mistakes. Price a missed buyer and a false alarm, and the optimal operating point is determined - it stops being a matter of taste.

• At Lumen the ratio is roughly 37 to 1 ($74 miss vs $2 false alarm), which pushes the optimal threshold far LOWER than 0.5: when misses are 37x dearer, you flag generously and tolerate noise.
• scikit-learn's own cost-sensitive example works exactly this way - a credit-risk case where one error is priced at five times the other, citing Elkan - and its tooling tunes the threshold to the money, not to a metric.
• Which means the real deliverable from the business side is not a metric target. It is two prices. Give your teams the cost of each mistake and the threshold stops being anyone's opinion.

Once you see the threshold, you see it everywhere. Each of these is the same dial - two mistake types with different prices, one cut line, and usually no named owner:

• Credit and risk approvals - approve a bad applicant vs decline a good one. The cut score is a threshold; ask who priced the two errors.
• Fraud and security alerts - a missed incident vs an analyst hour burned per false alarm. Alert fatigue is a threshold set too low, silence is one set too high.
• Content moderation - harmful content left up vs legitimate content taken down. Both mistakes make headlines; the dial decides which one you read about.
• Hiring screens - a strong candidate filtered out vs an interview slot spent on a weak one. Whoever set that cutoff made a talent-strategy decision.
• Lumen's outreach - tonight's case: a $74 miss vs a $2 nudge, 37 to 1.

The homework asks you to pick ONE of these in your own org and find the dial's owner. Expect the search to be short and the answer to be "nobody" - that is the finding.

Andrew Ng's doctrine for structuring ML projects is the cleanest fix a leader can mandate: split metrics into one optimizing metric and several satisficing metrics - thresholds to clear, not scores to maximize.

• The optimizing number should be the money (or the mission equivalent): expected cost or value per decision, using your two prices from Part 1.
• Everything else becomes a floor or a ceiling: recall at least X so finance's exposure is capped, precision at least Y so marketing's budget survives, response time under Z. Pass/fail, not leaderboards.
• This kills the Goodhart game structurally: nobody can win by polishing a proxy, because the only number being maximized is the one wired to the P&L.

One caution to carry forward: even the money number on an offline slice is a proxy for what happens in production - Google's "Rules of Machine Learning" makes this point as Rule #39, and it is the plot of a5 and a6. Tonight's fix is still the right first move: proxy-for-money beats proxy-for-nothing.

Four signs, any one of which deserves a meeting. Two or more and you are almost certainly funding metric-polishing:

• The model metric improves for months while the business line it serves stays flat. The churn story, generalized - the proxy and the money have decoupled.
• Nobody can state the metric's cost assumption. Ask "what does this metric assume the two mistakes cost?" F1 assumes 1 to 1; if nobody knows that, nobody checked whether it is true.
• The metric was chosen because it is standard, not because it matches your prices. "Everyone reports F1" is a fashion statement, not an evaluation design.
• The metric IS the team's target or OKR. That is Goodhart's precondition by definition - the fix is to move the target to the money and demote the metric to a floor.

• 1 · "What cost ratio does your problem actually have?" F1 assumes 1 to 1. If the honest answer is 37 to 1 - or anything far from even - the metric and the business are optimizing different worlds. A team that has never priced the two mistakes cannot know whether F1 is fine or catastrophic.
• 2 · "Who picked the threshold?" If the answer is "the default" or a name from the data team alone, the finance-vs-marketing trade from Part 1 was made without either party in the room. The dial needs an owner who knows both prices.
• 3 · "What does the cost-optimal point look like - and why aren't we there?" Tonight you saw it: at Lumen it looks terrible on metrics (F1 0.161) and saves $704 per slice. Sometimes there is a real reason not to operate there - capacity limits, customer-experience floors. Fine: those are satisficing constraints, and they should be stated, not hidden inside a metric choice.

Notice these are governance questions, not math questions. You are not auditing formulas - you are auditing whether a business decision was made by someone authorized to make it.