The cost matrix

Teams stall here: "we don't know the exact cost of a false negative." You do not need exact - you need the RATIO roughly right, because the optimal threshold moves with the ratio, not the absolute figures. Getting prices:

• FP price - usually operational: cost of the outreach, the analyst-minutes to triage an alert, the goodwill hit of a wrong takedown. Lumen: ~$2 a nudge.
• FN price - usually opportunity: the average order (~$74 from the Lumen canon), the average fraud loss, the downstream cost of the missed defect.
• Disagree about the number? Run the evaluator at BOTH candidate prices - if the chosen threshold barely moves, the disagreement doesn't matter; if it moves a lot, you found a genuinely important business question, and it belongs to the business owner, not the notebook.

sklearn's canonical example does exactly this with the German Credit dataset - a cost matrix where misclassifying bad credit as good is 5x more costly, shipped WITH the dataset since the 1990s. The field has always known; the defaults just made everyone forget.

Two tools, both in sklearn >= 1.5, both from the threshold-tuning user guide:

from sklearn.model_selection import TunedThresholdClassifierCV
from sklearn.metrics import make_scorer, confusion_matrix

def lumen_gain(y, y_pred):                 # negative cost = gain
    tn, fp, fn, tp = confusion_matrix(y, y_pred).ravel()
    return -(fp * 2 + fn * 74)

tuned = TunedThresholdClassifierCV(
    model, scoring=make_scorer(lumen_gain)
).fit(X_train, y_train)                    # NEVER tune on training data alone -
print(tuned.best_threshold_)               # it cross-validates internally

And for a threshold someone hands you (a policy decision, a regulator): FixedThresholdClassifier. The guide's two warnings, verbatim in spirit: default 0.5 cutoffs are "most certainly not ideal for most use cases," and never use the same data to train the classifier AND tune the threshold.

Andrew Ng's Structuring ML Projects doctrine, still the cleanest fix:

• Pick ONE optimizing metric - the number you genuinely try to maximize. Make it the money one when you can (expected cost/gain), a proxy only when you must.
• Make everything else a satisficing metric - a constraint with a floor or ceiling: recall >= 0.9, latency <= 200ms, cost per query <= $0.01. You don't optimize constraints; you respect them.
• His companion rule - the single-number evaluation metric - is about ITERATION SPEED, not truth: one number to rank experiments fast, with the cost matrix deciding what that number is.

Anthropic's own success-criteria guidance says the same thing for LLM systems: "most use cases need multidimensional evaluation" - their worked example is an F1 floor AND a toxicity ceiling AND a latency ceiling. One optimizing number, several guardrails. That shape repeats in b8.

Google's Rules of ML, Rule #39, generalizes the trap: launch metrics are themselves proxies. The full chain is offline eval score -> online metric -> long-term business goal, and each hop can break independently:

• Offline-to-online breaks via drift, serving skew (Rule #37 - b9's whole story), or a metric detached from user pain.
• Online-to-business breaks when engagement rises while satisfaction falls - the classic clickbait failure.

The defense is not a cleverer metric - it is knowing which hop you are standing on when you read a number, and never celebrating hop-one progress as hop-three victory. Tattoo for the capstone: my eval score is a proxy for a proxy.