Where this sits in the practitioner track
Session 3 found the leaked field and session 4 reproduced the developer's number from the data rather than believing it. Session 5 built a three-feature challenger and found the champion's real edge to be 0.006 of AUC. All three are questions about the model's construction. Tonight is the first test that goes outside the build entirely and asks what happened to the customers: the second of the three components of validation, and the one that produces the headline finding of the whole course. Session 7 takes the same comparison to a period the model never saw. The worked example is Brackwater throughout, with 2,000 later-vintage applications at a bad rate of 0.324.
★ What you walk out with tonight. One design rule strong enough to change every monitoring pack you touch: band every outcomes test, and state the banding, the population and the window next to the result. The Brackwater demonstration of why, where an aggregate gap of -0.006 sits on top of band gaps running from +0.068 to -0.142. The habit of keeping two different questions apart, because a model can rank customers acceptably and still put the wrong number on each one. And a working sense of what outlier analysis catches that no band average ever will.
LiveWhat outcomes analysis is, in the guidance and in practice4 min▶
The revised interagency guidance of 17 April 2026 defines outcomes analysis as the activity that "compares model outputs to corresponding real-world outcomes to assess model performance relative to model objectives and business use". Two halves of that sentence are worth separating. Comparing outputs to outcomes is the mechanic. Assessing performance relative to the model's objectives and its business use is the judgement, and it is the reason two models with identical error statistics can get different validation opinions.
The guidance is deliberately loose about the form the activity takes. It can be testing done during development, it can be an ongoing monitoring report, or it can be a standalone activity "such as back-testing or outlier analysis". A validator is therefore not obliged to invent a new test; a validator is obliged to make sure that a comparison of output against outcome exists somewhere, is done at a resolution capable of showing a problem, and reaches somebody who can act on it.
One clause is easy to skip and matters for models that are not pure statistics. Where a model "relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment". A judgemental overlay, an expert-set cutoff or a hand-tuned scaling factor is not exempt from this test. It is precisely the thing the test was written for, because expert judgement is the part of a model with no development statistics attached to it, and outcomes are the only evidence about it anybody will ever have.
Brackwater ran outcomes analysis every quarter for two years. The pack contained one line per model: predicted average default rate, actual average default rate, and the difference. For the scorecard the difference never exceeded a hundredth. The test was real, it was run on time, it was signed, and it was incapable of detecting what was wrong.
Self-studyFour words that get used interchangeably and should not be3 min▶
- Outcomes analysis. The component. Comparing model outputs against the corresponding real-world outcomes, in whatever form fits the model. Everything below is a way of doing it.
- Back-testing. Named in the guidance as one standalone form of outcomes analysis. Running the model over a historical period whose outcomes are already known and comparing what it would have said against what happened.
- Outlier analysis. Named in the guidance in the same breath. Looking at the individual cases the model got most wrong, rather than at the average of everybody.
- Ongoing monitoring. A separate component, not a synonym. It asks whether the model still performs as expected given changes in the world around it, and session 8 owns its design. Outcomes analysis supplies much of the evidence it consumes, which is why the two blur in conversation.
Keeping them apart matters in a report. "We perform outcomes analysis quarterly" tells a reviewer nothing. "We back-test quarterly by decile of predicted risk on the trailing twelve months, and review the twenty largest individual misses" tells them what could and could not have been detected.
The average was right and every band was wrong
11 min liveScore the champion on the later vintage, take the mean of what it predicted and the mean of what happened, and the two agree to within six thousandths. Now split those same 2,000 applications into five equal bands by how risky the model said they were, and take the same two means inside each band. Nothing about the data has changed; the only thing that changed is the resolution of the question. The model under-predicts its safest customers by up to 0.068 and over-predicts its riskiest by 0.142, which is a different model from the one the summary line described.
LiveWhy the errors cancel, and what that does to a monitoring pack4 min▶
This is not bad luck and it is not rounding. The model is systematically too optimistic about the customers it thinks are safe and systematically too pessimistic about the customers it thinks are risky. The first three bands come in above prediction, by +0.061, +0.068 and +0.035. The last two come in below, by -0.054 and -0.142. Sum across the whole book and the two directions offset to within six thousandths.
Now think about what that does to a report. A monitoring pack that carries one calibration figure per model per quarter will show this model as healthy for exactly as long as the two halves stay in balance. The balance is not a property of the model; it is a property of how many customers happen to sit in each band and how far each band is out. Both of those move with the book. So the pack shows green, green, green, and then on the quarter the halves stop offsetting it shows the model as suddenly broken, with no preceding signal and no explanation available from anything the pack ever recorded.
The failure mode to name in the report is not "the model is miscalibrated". It is "the test we run cannot distinguish a calibrated model from this one". That is a statement about the control, and controls are things a governance committee knows how to fix.
Eight quarterly packs at Brackwater reported a calibration difference under one hundredth, every time. The ninth would have been the first to look wrong, and by then the book had already shifted toward the band with the largest error. The pack was not wrong on any of the eight occasions. It was answering a question nobody should have asked on its own.
LiveThe design rule: band every outcomes test, and say how you banded it4 min▶
Banding is not extra credit and it is not a nice-to-have for the appendix. It is the only version of the test that can fail for the right reason. An unbanded comparison of two averages has one degree of freedom and can be defeated by any pair of offsetting errors, which in a ranking model is not an exotic circumstance but the normal shape of miscalibration.
Three things belong next to every banded result, and a result missing any of them is not reproducible by the next person to hold the report.
- The banding. Quintile or decile of predicted risk, not of any observed attribute. Banding on something the model did not produce answers a different question and usually a less useful one.
- The population. Which applications are in scope, and how many are in each band. A band with forty accounts in it will wander for reasons that have nothing to do with the model, and a reader who cannot see the count cannot tell the difference.
- The window. Which outcome period, and whether it has fully matured. An outcome window that has not closed makes every band look better than it will finish.
Decile rather than quintile is the usual choice once volumes allow it, and the trade is straightforward: finer bands localise the error better and make each band noisier. State which you chose. A validator who reports "gap by decile" without saying the deciles were cut on predicted risk over a twelve-month matured window has produced a number nobody can check.
Self-studyWhere an aggregate number is still the right thing to report3 min▶
The rule is to band every test, not to delete the aggregate. The aggregate answers a real question, which is whether the total expected loss the model implies for the book is close to the total that occurred. For provisioning, pricing at portfolio level and capital, that total is the quantity the business actually carries, and -0.006 on it is genuinely reassuring about that quantity.
What the aggregate cannot do is support any statement about an individual decision. Every decision this model drives is made one applicant at a time, at a point on the score distribution, and the error at that point is the one that matters to the applicant and to the credit decision. Reporting both, side by side, with the aggregate labelled as portfolio-level, is the honest presentation. Reporting only the aggregate is what happened here.
There is a second reason to keep it. When the bands stop offsetting, the aggregate will move sharply, and a validator who has both series can date the change. A validator who only ever kept bands has to reconstruct the total to explain what the business already noticed.
Ranking well and being right are not the same claim
10 min liveOutcomes analysis is usually run as if it were one test with one answer. It is at least two questions, and Brackwater is the clean case where they disagree. Does the model put riskier customers above safer ones is one question. Is the number it attaches to each of them right is a different one. A model can be acceptable on the first and badly wrong on the second, which is exactly what the band table shows, and a report that collapses them into a single verdict of "performance is satisfactory" has thrown away the finding.
LiveNaming the two questions without defining either metric3 min▶
A validator does not need to be able to derive a measure to use it as evidence, and reaching for a definition in a validation report is usually a sign that the finding has not been written clearly. What a validator needs is the question each measure answers and the class of defect it is blind to.
Discrimination asks about order. Given two customers, one of whom defaulted, how often does the model put the defaulter higher. Nothing in that question involves the number attached to either customer, which is why a model whose predictions were all divided by two would answer it identically.
Calibration asks about level. Among the customers the model said were 30 percent likely to default, did roughly 30 percent of them default. Nothing in that question involves the ordering, which is why a model that ranked no better than a coin toss but predicted the base rate for everybody would look excellent on the aggregate version of it.
How each of those is measured, what the standard statistics are, how to read the curves and where the thresholds come from is learn-model-evaluation-with-phoebe's territory and it is taught properly there. Borrow the numbers, cite the source, and keep the validation report about what the numbers are evidence for.
LiveWhat outlier analysis catches that a banded average does not4 min▶
The guidance names outlier analysis alongside back-testing as a standalone form of outcomes analysis, and it is the form that gets dropped first when time is short. It is worth defending, because banding and averaging are the same operation applied twice, and both of them destroy the evidence for a whole class of defect.
Consider a band of four hundred accounts whose average is exactly on target. That average is consistent with four hundred accounts each slightly off, and it is equally consistent with three hundred and eighty accounts exactly right and twenty accounts catastrophically wrong in the same direction as each other. Those two situations call for completely different responses, and no banded table anywhere in the pack can tell them apart.
The twenty are what outlier analysis looks for. In practice the things it surfaces are of four kinds, and all four are common. A data defect affecting one source, one branch or one channel. A segment the model was never fitted on and has no evidence about, which is the situation session 7 walks into. A feature taking a value outside anything in development, where the model is extrapolating rather than interpolating. And a genuine population the model simply handles badly, which is a limitation to write down rather than a bug to fix.
The operational form is unglamorous and works. Rank by the size of the individual error, take the largest twenty or fifty, and read them as cases rather than as rows. If a pattern appears across them, you have a finding that no aggregate would ever have produced. If nothing appears, you have spent an hour and can say in the report that you looked.
Self-studyOutcomes analysis when the model is substantially expert judgement3 min▶
The clause is short and it is the one most often skipped. Where a model "relies substantially on expert judgment, quantitative outcomes analysis helps to evaluate the quality of that judgment". Read it as an instruction about scope: the judgemental parts of a model are inside the boundary of the test, not outside it.
Most consumer scorecards carry at least one. A cutoff set by a credit committee rather than by an optimisation. A policy rule that declines an applicant regardless of score. A scaling factor applied to bring the model's output onto a business scale. An override rate where underwriters reverse the model. None of these has a development statistic, because none of them was fitted, and their absence from the model's performance section is usually treated as natural.
The test is the same test. Take the population the judgement applies to, compare what the judgement implied against what happened, and band it. An override population is the sharpest version: the model said one thing, a human said another, and the outcome is known. Comparing the outcome rate of overridden cases against the model's prediction for those same cases is outcomes analysis on the judgement itself, and it is frequently the single most informative table in a validation of a scorecard that has been in production for two years.
Produce the banded table and the number that hides it
18 min build-alongThe whole finding is about thirty lines of code, and the order matters: build the banded table first and derive the aggregate from it, rather than the other way round. A validator who computes the aggregate first has already anchored on -0.006 and will read the bands as a refinement of a good result instead of as a contradiction of it.
Score the model on the population you are testing, and keep the predictions. Not the fitted values from development. The comparison is only outcomes analysis if the outcome was unknown to the fit, which for Brackwater means the 2,000 later-vintage applications and not the 4,000 the model was estimated on.
Confirm the outcome window has closed before you compare anything. A partially matured window makes every actual rate look lower than it will finish, and the bands most affected are the ones with the longest time to default. This is the single most common way a banded table gets published wrong.
Cut bands on predicted risk, not on anything observed. Quintiles if the population is small, deciles if it is not. Record the cut points, because the next person to run this needs to reproduce your bands and quantile boundaries move with the sample.
Inside each band, take the mean prediction and the mean outcome, and print the count. Three columns, plus the signed gap. The count is not decoration: it is what tells the reader whether a band's gap is a finding or a sample-size artefact.
Sign the gap consistently and say which way round it is. Actual minus predicted throughout, so a positive number always means the model was too optimistic. Half the confusion in calibration reporting is two tables in the same pack using opposite conventions.
Only now compute the aggregate, and put it on the same page. -0.006 against band gaps from +0.068 to -0.142. Presenting them together is the finding; presenting either alone is a different and less useful report.
Rank the individual errors and read the top twenty as cases. The banded table has averaged away everything specific. Sort by the size of the individual miss and look at what is actually in there, because a concentrated segment inside an on-target band is invisible to every other step above.
Write the banding, the population and the window into the report itself. Not into a footnote and not into a methodology appendix nobody opens. A result whose banding is unstated cannot be challenged, and a result that cannot be challenged is not evidence.
import pandas as pd
# oot: the 2,000 later-vintage applications, already scored by the
# champion refit without the leaked field. oot["bad"] is the outcome.
oot = pd.read_csv("brackwater_oot_scored.csv") # columns: pred, bad
# Cut bands on PREDICTED risk, not on anything observed.
oot["band"] = pd.qcut(oot["pred"], 5, labels=[1, 2, 3, 4, 5])
banded = oot.groupby("band", observed=True).agg(
n = ("bad", "size"),
predicted= ("pred", "mean"),
actual = ("bad", "mean"),
).round(3)
# Sign it one way and keep it that way: positive means too optimistic.
banded["gap"] = (banded["actual"] - banded["predicted"]).round(3)
print(banded)
# band n predicted actual gap
# 1 400 0.086 0.147 0.061 <- under-predicts the safest
# 2 400 0.169 0.237 0.068
# 3 400 0.270 0.305 0.035
# 4 400 0.432 0.378 -0.054
# 5 400 0.694 0.552 -0.142 <- over-predicts the riskiestpredicted = oot["pred"].mean()
actual = oot["bad"].mean()
print(round(predicted, 3)) # 0.330
print(round(actual, 3)) # 0.324
print(round(actual - predicted, 3)) # -0.006
# Side by side, which is the only honest way to show either of them:
print("aggregate gap ", round(actual - predicted, 3)) # -0.006
print("worst band gap ", banded["gap"].min()) # -0.142
print("best band gap ", banded["gap"].max()) # 0.068
# Three bands are too optimistic and two are too pessimistic, so the
# five errors offset to within six thousandths across the whole book.
# Report the aggregate as a portfolio-level figure and never as a
# statement about any individual decision the model drives.Keep the band counts in the printed table even when they are all identical. They will not stay identical. The moment somebody filters to a segment, splits by channel or reruns on a shorter window, the bands stop being equal and a reader who has been trained by your earlier tables to ignore the count will misread the first uneven one.
Three questions
A monitoring pack has reported one calibration figure per model per quarter for two years, and Brackwater's scorecard has never been out by more than a hundredth. What is the correct finding?
The eight packs were arithmetically correct. Scored by quintile the same model is out by +0.061, +0.068, +0.035, -0.054 and -0.142, and the aggregate gap of -0.006 is what those five sum to. Writing this up as "the model is miscalibrated" invites a debate about the model; writing it up as "the control cannot fail for the right reason" is a statement about a control, which is a thing a committee can fix.
A band of 400 accounts has an average gap of almost exactly zero. What can you conclude about the accounts in it?
Banding and averaging are the same operation applied twice, and both destroy the evidence for a concentrated error inside a band. This is what outlier analysis is for, and the revised interagency guidance of 17 April 2026 names it alongside back-testing as a standalone form of outcomes analysis. Finer bands help, but no band average of any width separates a diffuse error from a concentrated one.
A model document states that a judgemental cutoff was set by the credit committee rather than fitted, and the performance section therefore reports nothing about it. How should a validator treat that?
The guidance puts the judgemental parts of a model inside the boundary of the test rather than outside it. The absence of a fitted statistic is the reason to run outcomes analysis on the judgement, not an excuse from it: take the population the judgement applies to, compare what it implied against what happened, and band it. An override population is the sharpest case, because the model said one thing, a person said another, and the outcome is known for both.
Before session 7
40 minFind the most recent monitoring pack for a model your team owns and answer one question about every performance line in it: at what resolution was this computed. Most lines will turn out to be a single figure over the whole book. Pick one of them, rebuild it by decile of predicted risk, and put the two versions on the same page with the band counts showing. Then rank the individual errors and read the top twenty as cases rather than rows. Bring both to session 7, which takes the same comparison to a period the model never saw and asks the harder question underneath it: whether the book being scored is even the same book the model was fitted on.