Uncertainty you can promise

After tonight the memo can carry two different creatures that both say "90":

• The human's trained 90 (session 2): "my 90 percent intervals capture the truth about 90 percent of the time - measured, on my own scored history." Certified by a hit rate.
• The algorithm's guaranteed 90 (tonight): "this prediction set contains the true label with 90 percent probability - guaranteed by construction, under stated conditions." Certified by a theorem.

They are not interchangeable, and the memo should never let them blur: the human number covers one-off judgment calls no model scores; the guaranteed number covers repeated, model-scored predictions. The memo carries both - labeled, so the reader always knows which certificate stands behind which number.

• Calibration curve - stated confidence on one axis, actual frequency on the other. The diagonal is honesty; session 2 drew yours, tonight draws the model's.
• Calibration set - labeled points the model never trained on, held out to price its errors.
• Nonconformity score - how "wrong" the model looks on one point, e.g. 1 minus the probability it gave the true label. High score = the truth surprised it.
• Coverage - how often the prediction set actually contains the truth. The thing being guaranteed.
• Exchangeability - the condition under which the guarantee holds: calibration data and new data behave like draws from the same well-shuffled deck. Distribution shift breaks it.

Three moves, all bookkeeping: take a batch of scored predictions with known outcomes, group them into confidence buckets (all the "about 80%" predictions together), and compute the actual positive rate inside each bucket. Plot stated against actual. A calibrated model sits on the diagonal; the sagging curve above shows the common failure - the high-confidence buckets are where overconfidence concentrates, exactly as it did in the human lab.

The connection to session 2 is the point: a model's "84% churn risk" is a confidence claim from an estimator, and it deserves the same audit a colleague's "90% sure" got. Nobody at Mango Lane had ever bucketed the churn model's outputs. The curve is an afternoon of work and it changes what the number is worth in the memo.

Ranking well and stating honest probabilities are different skills. A model can order customers from safest to riskiest almost perfectly while its stated probabilities run systematically hot or cold - training objectives reward separation, not honesty, and modern flexible models tend toward overconfidence at the extremes. Two practical consequences:

• Never read a raw score as a probability until the calibration curve has been drawn on held-out data. The curve is the model's session-2 lab result.
• Calibration drifts. A curve drawn last year certifies last year. Re-audit on a cadence, and treat a sudden sag as an early warning that the world the model learned has moved - which is exactly the condition that voids tonight's guarantee too.

Split conformal is four moves, and only one involves any math (paraphrasing the Angelopoulos-Bates tutorial):

• Hold out a calibration set - labeled points the model never saw in training. This is the model's exam, not its homework.
• Compute a nonconformity score per calibration point - a number for how wrong the model looks there, e.g. 1 minus the probability it assigned the true label.
• Take the ceil((n+1)(1-alpha))/n empirical quantile of those scores - for 90% coverage, alpha is 0.1. This threshold is where the guarantee is manufactured: the (n+1) is a finite-sample correction, not a typo.
• Build the prediction set for a new customer: every candidate label whose score falls below the threshold goes in.

That is the whole method - a few lines on top of ANY model, including the gradient booster you built in the ensemble course. Uncertain customers naturally get bigger sets, confident ones get singletons; the set SIZE becomes an honest per-customer uncertainty readout even though the guarantee itself is an average.

Two conditions bound the promise, and the memo must carry both:

• Marginal, not conditional. Coverage holds ON AVERAGE over exchangeable data - roughly 90% of customers get a set containing their truth. It is NOT a per-instance promise: no individual customer is personally guaranteed 90%, and coverage can run lower on some slices while the average still holds.
• Exchangeability, not forever. The theorem assumes calibration data and new data are exchangeable - same well-shuffled deck. Distribution shift breaks that: a drifted quarter voids the promise outright.

The second condition is not a weakness to whisper - it is a reason to use session 1's machinery. The memo's flip-trigger should name it: "these sets are guaranteed under stable conditions; if the input distribution shifts materially, recalibrate before trusting them." A guarantee with stated conditions is a promise; one with hidden conditions is a liability.

The upgrade is auditability. A CFO cannot audit "high risk". They can audit a hit rate, and they can audit a coverage guarantee with stated conditions - which means the estimates section stops being a trust exercise and becomes a checkable document. That is what the promise buys.