Where this sits in the practitioner track
Session 6 compared what the model said against what happened, and found an aggregate calibration gap of -0.006 sitting on top of band gaps from +0.068 to -0.142. That test already used the later vintage, and it used it without asking the obvious question: why is this vintage behaving differently at all. Tonight asks it. Out-of-time testing measures how much performance the model loses on a period it never saw, and population stability tells you whether the loss has an explanation in the book or has to be explained some other way. Session 8 designs the monitoring that would have caught this before a validator did. The worked example is Brackwater, with 4,000 development applications and 2,000 from the following year.
★ What you walk out with tonight. A measured out-of-time result on a model whose truth is known: 0.772 on development, 0.682 on the later vintage, a degradation of 0.090. The discipline of reporting a population comparison beside every out-of-time number, because a degradation with no accompanying population evidence cannot be acted on. The three-way distinction that actually decides what happens next, between recalibration, an overlay and redevelopment. And the habit that separates a validator from a reporter of numbers: writing the threshold down before it is crossed, rather than deciding afterwards whether the movement mattered.
LiveOut of sample and out of time are two different guards4 min▶
Both are named in the current guidance as part of model testing, and they are routinely treated as the same exercise done twice. They are not. Out-of-sample testing holds back rows from the same period and asks whether the model has fitted noise that does not repeat. Out-of-time testing holds back a whole later period and asks a bigger question: whether the relationship the model learned is still the relationship that governs the world it now runs in.
A model can pass the first and fail the second, and Brackwater is that case. Nothing in the development sample suggested a problem. Split it any way you like and the model looks stable, because every row in it comes from the same year, the same economy and the same mix of applicants. The later vintage is drawn after all three of those have moved, and the model loses 0.090 of its discrimination on it.
This is also why the order of a validation matters. An out-of-time test run before the population comparison produces a number with no explanation attached, and a number with no explanation attached gets argued about rather than acted on. Run both, and report them together.
Brackwater's model document contained an out-of-sample section and no out-of-time section, on the reasonable-sounding grounds that there was no later data at build time. Two years later there was a great deal of later data and nobody had gone back. The test was never refused; it was deferred once and then inherited by people who did not know it was outstanding.
Self-studyFour things that all get called "the model degraded"3 min▶
- The population moved. The mix of applicants changed, but the relationship between features and outcome is the same. A model fitted on one mix and scored on another loses performance without anything about it being wrong.
- The relationship moved. The same applicant profile now defaults at a different rate, or a feature that used to carry signal carries less of it. This is the deeper failure, and it is the one that recalibration does not repair.
- The outcome moved. The base rate changed, here from 0.292 to 0.324. That alone shifts every predicted level without touching the ordering, which is why it shows up in calibration long before it shows up in discrimination.
- The pipeline moved. A feed changed, a definition was revised, a join started returning nulls. This looks exactly like the other three from a performance chart and is the first thing to rule out, because it is the only one where the fix is not a modelling decision.
Detecting these shifts as a technique, with the statistics and the continuous evaluation machinery behind them, is learn-model-evaluation-with-phoebe's territory. A validator measures the movement, names which of the four it is consistent with, and says what follows. Those are different jobs, and only the second one belongs in a validation report.
Score it on a year it never saw
11 min liveThe mechanic is unremarkable. Take the model as fitted on development, change nothing about it, and score the later vintage. Do not refit, do not retune, do not reselect features; the point of the test is to measure what the deployed object does on new data, and any adjustment made along the way measures something else. The champion refit without the leaked field scores 0.772 on development and 0.682 on the later vintage. The interesting part is not the 0.090 itself but what is available to explain it.
LiveThree causes, three different remediations4 min▶
The reason to separate the three movements is that they do not lead to the same action, and a validation that reports only the 0.090 leaves the model owner to guess which action was meant.
The mix of borrowers changed. Thin-file applicants went from 19.0 percent of the book to 34.0 percent. The model was fitted on a population where they were a minority and is now scoring a population where they are a third. It has less evidence about them than about anybody else, and it is now being asked about them twice as often. This is a coverage problem, and the honest responses are to restrict the mandate, to add an overlay for that segment, or to refit once enough of them have matured outcomes.
A relationship changed strength. The utilisation relationship is not the same in the later vintage as in the development one. A coefficient fitted on the old strength is now the wrong coefficient, and no amount of rescaling the output fixes a weight on an input. This is the case that points at redevelopment, and it is the case model owners resist hardest because it is the most expensive.
Something moved that has no column. An economy-wide shock is not in the feature list and cannot be. The model has no way to know about it and no way to adjust for it, and the bad rate moving from 0.292 to 0.324 is partly this. The response here is an overlay or a management adjustment, applied on top of the model rather than inside it, documented as a judgement and then itself subject to outcomes analysis.
Recalibration is the fourth answer and the most commonly reached for. It repairs levels, not ordering: if the ranking is intact and only the rates attached to it are wrong, recalibrating is the proportionate response. It does nothing at all for a relationship that has changed strength, and applying it there buys quiet numbers for a quarter or two while the underlying problem grows.
LiveScore it unchanged, or you have measured the wrong thing3 min▶
The commonest way an out-of-time test gets quietly invalidated is that somebody refits on the way. It usually happens for good reasons: a feature no longer exists in the later data, a category has new levels, a scaler was rebuilt because the code path expected it. Each of those is a small accommodation and each one converts the test from "what does the deployed model do on new data" into "what could a model like this do if it were rebuilt", which is a question nobody asked.
The test to apply is simple. Every coefficient, every cut point, every encoding and every imputation rule must be the ones that were in force at the end of development. If something cannot be held fixed because the later data will not support it, that is itself a finding and belongs in the report, because it means the deployed model cannot be scored on current data without intervention.
The second invalidation is more subtle and involves the outcome window. The later vintage must have matured outcomes, and a vintage whose window is still open will flatter the model, because the defaults that have not happened yet are counted as non-defaults. A validator who does not state the maturity of the out-of-time window has not stated whether the result is a floor or an estimate.
Self-studyHow large a drop is a finding3 min▶
There is no universal threshold and anybody offering one should be asked where it came from. What there is instead is a defensible way to set one for a given model, and it rests on three things the validator can actually establish.
The first is the sampling variability of the measure on a sample of this size. A drop inside the range you would see from resampling alone is not evidence of anything, and 2,000 applications is not a large sample. The second is the business consequence at the cutoff in use, which is a far better currency than the measure itself: session 5's comparison is in approval rates and bad rates among approved, and those translate directly into a number the committee already understands. The third is what the model is used for, since the guidance ties the rigour of validation to the model's approach, use and materiality.
The thing that is not defensible is deciding after the fact. A threshold chosen once the number is known is a rationalisation, and everyone in the room can tell. Write the threshold down before the test runs, state what happens when it is crossed, and accept the consequence if it is. Session 8 is where this becomes a standing control rather than a one-off judgement.
Is it even the same book
10 min liveA degradation number on its own is a symptom with no diagnosis. The population comparison is what turns it into one, and it is cheap: means and shares for the features the model uses, development against the later vintage, side by side. At Brackwater it says something specific. The share of thin-file applicants has nearly doubled, prior delinquency is up modestly, mean utilisation is up slightly, and the bad rate has risen from 0.292 to 0.324. The book moved toward exactly the borrowers the model has least evidence about, which is a sentence a validator can put in a report and a committee can act on.
LiveA population shift is not a failure, it is a reason to expect one4 min▶
This is the sentence to take away. A book that has changed shape has not done anything wrong, and neither has the model. Lending books move because businesses grow, channels open, competitors withdraw, marketing changes and the economy turns. A validator who treats every shift as a defect will be ignored within two cycles, and correctly so.
What a shift does is change the standing of the evidence. The development testing in the model document was evidence about a population that is 19.0 percent thin-file. It is weaker evidence about a population that is 34.0 percent thin-file, and it is weaker in a specific, nameable way: the segment now carrying a third of the volume was a minority of the rows the coefficients were estimated from. Nothing about the model changed. The question the evidence answers stopped matching the question being asked.
So the validator's job here has two halves. Say which shifts the model can absorb, which for most models means shifts inside the range the development data covered, where the model is interpolating. And say which it cannot, which means shifts that take a segment outside that range or that make a sparse segment material. Then write the threshold down before it is crossed, because the value of this analysis is entirely in having done it early.
The thin-file share at Brackwater did not jump. It drifted, a little each quarter, through a period when the business was deliberately opening a newer-to-credit channel and reporting that as a success. Every individual quarter's movement was defensible. No individual quarter's movement crossed anything, because nothing had been written down to cross.
LiveWhich shift the model can absorb, and writing it down first4 min▶
A threshold that is useful has four parts, and a threshold missing any of them will not survive its first breach.
- The quantity. Named precisely enough to be recomputed by somebody else, with its population and its window. "Thin-file share of booked applications, trailing three months" is a quantity. "Population stability" is a topic.
- The level. A number, set with a stated reason. The reason does not have to be sophisticated, and "this is the edge of the range the model was fitted on" is a better reason than most of the sophisticated ones.
- The action. What happens on breach, specifically. Not "escalate", but who is told, what they are required to produce, and by when. A trigger whose action is a notification is a trigger that fires into a mailbox.
- The owner. A named role that carries the consequence. Without this, the breach gets discussed and the model keeps running, which is the observed outcome in most institutions.
The reason to write this before the number is known is that nobody can do it afterwards. Once the thin-file share reads 34.0 percent, any threshold you propose is a position in an argument. Proposed at 19.0 percent, when nothing is at stake and the channel is new, the same threshold is uncontroversial and takes ten minutes to agree. The whole value of the control is spent in that ten minutes, and session 8 is about not wasting it.
Self-studyWhat the guidance says happens next3 min▶
Ongoing monitoring is the third component of validation, and the current text defines it as "an evaluation of the extent to which a model is performing as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions". That list is worth reading item by item against Brackwater, because three of the six are directly in evidence tonight. Clients, in the shift toward thin-file applicants. Data relevance, in development evidence that is now about a different mix. Market conditions, in the economy-wide shock with no column for it.
The guidance goes on to say that a model no longer performing as expected may warrant overlays, adjustment or redevelopment. Three named options, in ascending order of cost, and the choice between them is exactly the three-way diagnosis from Part 1. An overlay for the shock nobody has a column for. Adjustment, in the sense of recalibration, for levels that have drifted while the ordering holds. Redevelopment for a relationship that has changed strength.
Tonight is the test that tells you one of those is needed. It does not design the control that would have told you sooner, and it does not decide the thresholds that make the control fire. Session 8 owns both. The reason they are separate sessions is that they are separate skills: measuring a shift is a technical exercise, and designing a control that somebody will actually act on is an organisational one.
Two tables, and neither is useful without the other
18 min build-alongThe out-of-time score is four lines of code and the population comparison is six. Together they are the whole of tonight, and the discipline is that they go in the report as a pair. A degradation with no population evidence is a number the model owner will explain away; a population comparison with no degradation is a table nobody reads.
Freeze the model before you go anywhere near the later data. Fit on development, then treat the fitted object as read-only. Every coefficient, cut point, encoding and imputation rule stays as it was. If your code cannot score the later vintage without changing one of them, stop and write that down, because it is a finding about the deployed model rather than a technical inconvenience.
Confirm the later vintage has matured outcomes. An open window counts defaults that have not happened yet as non-defaults and flatters the model. State the maturity in the report so the reader knows whether your number is a floor or an estimate.
Score it unchanged and report both numbers with the degradation named. 0.772 and 0.682, and the 0.090 between them. Reporting only the second one invites the response that it was never that good; reporting only the difference hides the level the business is actually running at.
Compare the population on every feature the model uses, not just the interesting ones. Means for continuous features, shares for indicators, and the outcome rate alongside them. The feature you did not check is the one the model owner will point at.
Rank the shifts by size and look at the largest first. At Brackwater the thin-file share moves by several times more than anything else, which makes it the sentence the report leads with. Prior delinquency at +2.8 points and utilisation at +0.016 are context, not headlines.
Say which of the four causes each shift is consistent with, and stop there. Mix, relationship, outcome level, or pipeline. Consistent with is the right strength of claim; a comparison of two populations does not identify an effect, and reaching for machinery that would is learn-causal-inference-with-phoebe's territory and unnecessary here.
Name the remediation the diagnosis points at, with its cost. Overlay, recalibration or redevelopment. The model owner will choose, and a validator who names all three with what each does and does not repair has made that choice an informed one rather than a budget one.
Propose the threshold now, while the number is uncontroversial. Quantity, level, action, owner. Written into the report as a recommendation, not left for the monitoring redesign, because the monitoring redesign is six months away and the book is still moving.
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
dev = pd.read_csv("brackwater_dev.csv") # 4,000 development applications
oot = pd.read_csv("brackwater_oot.csv") # 2,000 from the following year
# The honest champion: refit without the leaked collections field.
feats = ["util", "priorDelq", "employed", "incomeRat", "thin",
"age", "n0", "n1", "n2", "n3"]
model = LogisticRegression(max_iter=2000, C=1e4).fit(dev[feats], dev["bad"])
# Fitted once, scored twice. Nothing is refitted on the later vintage.
dev_auc = roc_auc_score(dev["bad"], model.predict_proba(dev[feats])[:, 1])
oot_auc = roc_auc_score(oot["bad"], model.predict_proba(oot[feats])[:, 1])
print(round(dev_auc, 3)) # 0.772 development
print(round(oot_auc, 3)) # 0.682 later vintage
print(round(dev_auc - oot_auc, 3)) # 0.090 the degradation
# If this block had to touch `feats`, a scaler or an encoder to run the
# second line, the test would be measuring a rebuild and not the model.def profile(df):
"""Shares for the indicators, means for the rates, outcome alongside."""
return pd.Series({
"thin-file share": df["thin"].mean(),
"prior delinquency": df["priorDelq"].mean(),
"mean utilisation": df["util"].mean(),
"bad rate": df["bad"].mean(),
})
compare = pd.DataFrame({
"development": profile(dev),
"later vintage": profile(oot),
}).round(3)
compare["shift"] = (compare["later vintage"] - compare["development"]).round(3)
print(compare.sort_values("shift", key=abs, ascending=False))
# development later vintage shift
# thin-file share 0.190 0.340 0.150 <- nearly doubled
# bad rate 0.292 0.324 0.032
# prior delinquency 0.340 0.368 0.028
# mean utilisation 0.448 0.464 0.016
# One shift is several times the others. The book moved toward exactly
# the borrowers the model has least evidence about, and the development
# testing in the model document is weaker evidence about that book.Put the two tables on one page and write one sentence between them. "Discrimination fell by 0.090 on the later vintage, over a period in which the thin-file share of the book rose from 19.0 percent to 34.0 percent." That sentence is the finding. Everything above it is evidence and everything below it is remediation, and a report structured that way is one a model owner can disagree with precisely rather than generally.
Three questions
Brackwater's champion, refit without the leaked field, scores 0.772 on development and 0.682 on the later vintage. What is the defensible reading of the 0.090?
Three things moved between the vintages by construction: the thin-file share of the book, the strength of the utilisation relationship, and an economy-wide shock with no column for it. Each points at a different remediation, and recalibration in particular repairs levels rather than ordering, so it does nothing for a relationship that has changed strength. Reporting the 0.090 without the diagnosis leaves the model owner to guess which action was meant.
The thin-file share of the book has gone from 19.0 percent to 34.0 percent. A model owner says no threshold was breached, so there is nothing to report. What is the flaw?
A population shift is not itself a failure, it is a reason to expect one. The drift was gradual and each quarter's movement was individually defensible, which is exactly how a book arrives somewhere nobody intended. A threshold proposed at 19.0 percent costs ten minutes to agree; the same threshold proposed at 34.0 percent is a position in an argument. Option B invents a level with no stated reason, which is the thing to avoid.
Under the revised interagency guidance of 17 April 2026, what does ongoing monitoring evaluate, and what may follow when a model is no longer performing as expected?
Three of the six named changes are directly in evidence at Brackwater: clients, in the shift toward thin-file applicants; data relevance, in development evidence that is now about a different mix; and market conditions, in the shock with no column for it. The three remedies are named in ascending order of cost and map onto the three-way diagnosis. Note that SR 11-7 is superseded; the April 2026 text is the current one. Governance frameworks are a different subject and belong to learn-ai-governance-with-phoebe.
Before session 8
45 minTake a model your team runs and build the population comparison for it: every feature the model uses, development against the most recent complete quarter, with the outcome rate alongside and the shifts ranked by size. Most of the work is finding the development profile, and if it is not recoverable that is your first finding. Then pick the largest shift and draft a four-part threshold for it: the quantity with its population and window, the level with a stated reason, the action on breach, and the named owner. Bring the draft to session 8, which is where thresholds stop being a validator's recommendation and become a control that fires.