Where this sits in the practitioner track
Session 6 compared predicted against actual by band and found the aggregate hiding the bands. Session 7 scored the frozen model on a year it had never seen, watched discrimination fall from 0.772 to 0.682, and found a book that had moved toward exactly the borrowers the model has least evidence about. Both of those were validator tests, run once, by somebody with a mandate and a deadline. Tonight they become a standing control that runs when nobody is looking, which is a different design problem with a different failure mode: a validator who finds nothing has done a bad review, and a monitoring pack that finds nothing looks like good news. Session 9 turns what fires into a finding somebody has to answer. The worked example is Brackwater's consumer scorecard throughout, 4,000 development applications at a bad rate of 0.292 and 2,000 from the following year at 0.324.
★ What you walk out with tonight. A test for whether a monitoring metric is worth reporting: can it fail for the right reason, on this model, given what you already know moves. The four parts every trigger needs before it counts as a control rather than a chart. A worked monitoring specification for Brackwater with seven rows, three of which fire on the later vintage and four of which do not. And the boundary of the whole technique, which matters more than any of it: monitoring compares the model against the world, so it is structurally blind to a model that is wrong in a way the world confirms.
LiveWhat ongoing monitoring is, and what it obliges4 min▶
Ongoing monitoring is the third of the three components of validation, and the revised interagency guidance of 17 April 2026 defines it as "an evaluation of the extent to which a model is performing as expected given potential changes in products, exposures, activities, clients, data relevance, or market conditions". Read the list of things that might have changed: products, exposures, activities, clients, data relevance, market conditions. Not one of them is a property of the model. Every one of them is a property of the world the model was fitted to, which is the clue to what the pack should contain.
A model that is no longer performing as expected "may warrant overlays, adjustment, or redevelopment". That sentence is doing more work than it looks like. It names three different repairs of three very different costs, which means the monitoring result is expected to be specific enough to choose between them. A pack that can only say "red" cannot.
An effective plan also includes "regularly assessing any model limitations at the development stage and over time, along with procedures for responding to any issues". The limitations are written at development time, before anything is running, and the procedure for responding is written then too. And the frequency and scope of monitoring reports depend on the nature of the model, the availability of new data or modelling approaches, and model materiality. Frequency is a risk decision, not a calendar convenience.
Brackwater's monitoring pack had one page per model: a population count, an approval rate, and a single calibration number. It was produced quarterly, circulated to eleven people, and had never once produced an action item in two years. That is not a pack nobody read. It is a pack that could not say anything.
Self-studyMonitoring, revalidation and a validator's test are three different things4 min▶
- A validator's test. Run once, by somebody independent, against a question they chose. Sessions 3 to 7 were all of these. It can be as expensive and as bespoke as the question deserves, because it happens once.
- Ongoing monitoring. Runs on a schedule, unattended, against questions chosen in advance. It has to be cheap enough to run every quarter forever and specific enough that a breach means something. Those two pressures pull in opposite directions and the design is the compromise.
- Periodic revalidation. The whole review done again, on a cycle set by materiality. Monitoring is what tells you whether the cycle is fast enough, and a monitoring trigger is what pulls a revalidation forward.
- An overlay. An adjustment applied to model output while the underlying model is unchanged. It is a repair, not a monitor, and it is the cheapest of the three repairs the guidance names. A model running on an overlay that nobody has revisited is its own finding.
Keeping them apart matters because the common failure is to promise the second and deliver nothing, on the grounds that the first was thorough. A thorough validation in March says nothing about September, which is the entire reason the third component exists.
Design the metric so it can fail
13 min liveThe usual way a monitoring pack gets built is to list the metrics somebody can compute cheaply and put them in a table. The better way is to start from what you already know can move on this model, and design each metric so that when the thing moves, the number moves too. Brackwater is the perfect teaching case because the evidence is already in hand: session 6 established that the errors are large, systematic and opposed, and session 7 established which part of the population shifted. Any metric that survives both of those facts and still reads as healthy is a metric that was never going to say anything.
LiveThe five things worth watching on this model, and where each came from5 min▶
None of these is on the list because it is a standard metric. Each one is on the list because something in sessions 3 to 7 showed it capable of moving on this model, which is the only defensible way to build the list.
- The banded outcome comparison, never the aggregate. Predicted against actual, by quintile of predicted risk, every band reported. Session 6 is the whole argument: -0.006 in aggregate over band gaps of +0.061, +0.068, +0.035, -0.054 and -0.142. The aggregate row can stay in the pack for continuity, as long as nobody is allowed to act on it alone.
- The population fields that actually drive the score. Thin-file share, which moved from 19.0 to 34.0 percent. Prior delinquency, 34.0 to 36.8 percent. Mean utilisation, 0.448 to 0.464. Three fields, because those are the three the development testing was thinnest about, not because three is a nice number.
- Discrimination on a recent window. Development gave 0.772 for the honestly refit champion and the later vintage gave 0.682. A monitoring metric here needs a window with matured outcomes, which is why this row runs on a slower cycle than the population rows.
- The bad rate itself. 0.292 on development and 0.324 out of time. It is not a model metric at all, and that is exactly why it belongs: it separates a model that has degraded from a book that has got riskier, and those two have different repairs.
- The approval rate and the bad rate among approved. Not a model metric either. It is what the business actually experiences, and a monitoring pack with no business row is a pack the business has no reason to read.
LiveWhy the aggregate survives in every pack you will ever see4 min▶
It is worth being fair to the aggregate, because the reasons it dominates monitoring packs are good reasons and it will lose the argument faster if you pretend otherwise. One number per model fits on one row. One row per model fits fifty models on a page, and fifty models on a page is what a committee can actually look at. The aggregate is stable, cheap, comparable across models and across quarters, and it does not need a decision about how many bands or where to cut them.
The problem is not that the aggregate is a bad number. It is that the aggregate is the average of the thing you care about, and averaging is precisely the operation that destroys the signal here. Under-prediction at the safe end and over-prediction at the risky end are the signature of a model whose relationship has moved, and it is a signature made entirely of opposing errors. Averaging opposing errors is how you get to -0.006.
So the practical design is not to delete the aggregate. It is to make the worst band the row that carries the threshold, and let the aggregate sit next to it as context. The committee still gets one number per model. It is just a number that can go red.
The one-line test for any proposed monitoring metric. Name a specific way this model could go wrong, then ask whether this metric would move if that happened. If you cannot name the failure, you do not yet know what the metric is for. If you can name it and the metric would not move, the metric is decoration and putting it in the pack is worse than leaving it out, because it will be cited as evidence of health.
Self-studyFrequency follows materiality, not convenience4 min▶
The frequency and scope of monitoring reports depend on the nature of the model, the availability of new data or modelling approaches, and model materiality. Every one of those three is a property of the situation, and none of them is the reporting calendar. In practice almost every monitoring pack in existence is quarterly, and almost none of them can explain why.
The nature of the model sets the floor on how fast anything can be known. Brackwater's outcome window means a default takes time to appear, so a discrimination row cannot be refreshed monthly no matter how material the model is: there is nothing new in the data. The population rows have no such constraint, because an application's features are known on the day it arrives. That alone splits the pack into two cycles, and a pack that runs everything on one cycle is either late on the fast rows or empty on the slow ones.
Materiality sets the appetite. A model driving the approval cutoff on a consumer book is a different proposition from one ranking leads for a call list, and the tiering call from session 1 is where that judgement was already made and written down. Reusing it here is the point of having made it. The availability of new data or modelling approaches is the one people forget: if a materially better approach became available, that is a reason to look again even when every row in the pack is green.
A threshold with no stated action is an observation
11 min liveChoosing what to watch is half the job and the easier half. The half that decides whether the pack does anything is the part after the number: what level counts as a breach, over what population and window, who is named, and what happens. Four parts. A row with a level and nothing else produces a chart that goes up and down and a meeting that notices. A row with all four produces a consequence somebody agreed to in advance, which is the only kind of consequence that survives contact with a busy quarter.
LiveWhere the level comes from, given that nothing sets one for you4 min▶
No supervisory text sets a number. There is no threshold in the guidance, there is no industry constant, and anybody who tells you the level is 0.05 because it is always 0.05 is quoting a habit. The level is a choice, it is the bank's choice, and the only thing that makes it defensible is that the reasoning is written down next to it.
Three ways to reason about it, in rough order of how well they survive being questioned. The first is consequence: how far can this quantity move before the decision the model drives becomes a decision you would not have made. That is the best argument and the hardest work, because it needs the business to say what it would do differently. The second is historical variation: how much does this quantity move quarter to quarter when nothing is wrong, so that a level can sit above the noise. That needs history, which a new model does not have. The third is the development baseline plus a stated tolerance, which is weak reasoning but honest, and it is what most first versions are.
What matters more than which of the three you use is that the level is attached to a baseline you can point at. Brackwater's baselines are the development figures the model was approved on: thin-file share 0.190, prior delinquency 0.340, mean utilisation 0.448, discrimination 0.772, bad rate 0.292. A trigger that says "moves by more than five points from the number in the approved model document" is a sentence anybody can check. A trigger that says "deteriorates materially" is not a trigger.
LiveWrite the triggers before go-live, because afterwards it is a negotiation4 min▶
The sequencing point is the most practically useful thing tonight and it costs nothing to apply. Before a model goes live, the level is an abstract question and everybody in the room can afford to be principled about it. After a breach, the level is a question about whether a specific model that a specific team built and a specific business depends on has to be taken off the road this specific quarter. The same number, asked in two different rooms, and the second room has a stake in the answer.
This is why the validation report is the right place to propose the thresholds, and why session 7 ended by telling you to draft one while the number was uncontroversial. A threshold written into a report that has been signed is a level that was agreed when nobody knew which way it would break. A threshold written after the pack turns red is an agreement about that particular quarter, and it will be revisited the next time it is inconvenient.
The same logic applies to the consequence. Deciding in advance that a breach of the worst-band calibration trigger means recalibration is a cheap decision. Deciding it in the meeting where the breach has just been reported means deciding it against a person who will explain, accurately, that recalibration takes eight weeks and the quarter ends in three.
Brackwater's pack had no thresholds at all, which is the most common arrangement and the one that produces the fewest arguments. Every number was reported as a number. The calibration gap of -0.006 was true, correctly computed, and circulated to eleven people, none of whom had been given a level against which to read it.
Self-studyWhat monitoring cannot see, and why session 2 comes first4 min▶
Ongoing monitoring compares the model against the world. That is its definition and its entire mechanism: is the model performing as expected given what has changed in products, exposures, clients, data relevance or market conditions. Which means the one defect it can never detect is a model that is wrong in a way the world confirms.
Brackwater is the example, and it is a clean one. The leaked collections field is a count of contacts logged after the outcome window opens. Run the champion as submitted in production, monitor it forever, and every row in the pack stays green: the field keeps being populated, it keeps being related to default at the same strength, the discrimination it produces keeps reproducing, and the model keeps predicting default in people who are about to be recorded as defaulting. There is no quarter on which that stops working. The relationship is real. It is a relationship with the future, and outcomes keep confirming it because it was built out of outcomes.
No threshold catches this. No banding catches it, no population comparison catches it, no amount of frequency catches it. The only thing that catches it is somebody reading the feature list and asking when each value is written, which is conceptual soundness, which is the first component and session 2 of this track. That ordering is not an accident of curriculum design. A monitoring programme built on top of an unexamined model is an expensive way to confirm that the model is consistent with itself.
The general form, worth carrying past credit scoring: monitoring answers "has anything changed since we approved this". It cannot answer "was the thing we approved ever right". Those are different questions and only one of them gets easier with more data.
Write the specification, then run the quarter against it
18 min build-alongA monitoring specification is a table, and the discipline is entirely in which columns it has. Metric, band, level, window, owner, action. Six columns, and a row that cannot fill all six is not ready to be a control. Build the table first, with no data anywhere near it, then score the later vintage against it and read out what fires. Doing it in that order is the whole exercise: a spec written while looking at the data is a spec that agrees with the data.
List what can go wrong with this specific model, before listing any metric. Three or four sentences, each naming a failure: the relationship moves, the book shifts toward a segment with thin evidence, outcomes arrive later than assumed, an upstream field changes meaning. The metrics come from this list, and a metric with no failure attached to it does not go in the pack.
For each failure, pick the quantity that would move if it happened. Then check the direction and the size you would expect. If the honest answer is "it would move a bit, probably", the metric is too blunt for the failure and you need a band, a segment or a different quantity.
Band anything that can be wrong in opposite directions at once. Calibration is the standing example and Brackwater is the proof: -0.006 in aggregate over band gaps from +0.061 to -0.142. Put the worst band on the threshold and leave the aggregate next to it as context, never as the control.
Write the baseline each level is measured from, and say where it came from. The development figures in the approved model document, quoted exactly. A trigger measured against last quarter drifts with the thing it is supposed to detect, which is how a book can move fifteen points without any row ever turning red.
Set the level, and write the sentence justifying it in the same row. Consequence-based if the business will tell you what it would do differently, historical-variation-based if you have the history, baseline-plus-tolerance if you have neither. Any of the three is defensible. The absence of a written reason is not.
Give each row a window that respects how fast the data can actually know. Population rows can run as fast as applications arrive. Anything that needs a matured outcome runs slower, and a row refreshed faster than its outcomes mature is reporting an artefact of incompleteness.
Name a person and pick the action from the fixed list. Investigate, overlay, recalibrate, redevelop, withdraw. A person, not a team, not a function, not the committee. The list is short on purpose: a free-text action column fills up with "review at next cycle", which is the written form of doing nothing.
Run the spec against a period you already understand, and check that something fires. A spec that passes everything on a quarter you know was bad is broken, and you will not find that out later, because a silent control and a healthy model look identical from the outside. Brackwater's later vintage is the test case: three of seven rows should fire.
import pandas as pd
# Six columns. A row that cannot fill all six is not a control.
# No supervisory text sets a level: these are Brackwater's choice,
# written into the validation report before the model went live.
spec = pd.DataFrame(
[
("calibration gap", "all applications", 0.050, "quarterly", "model owner", "investigate"),
("calibration gap", "worst quintile", 0.050, "quarterly", "model owner", "recalibrate"),
("thin-file share", "all applications", 0.050, "quarterly", "model owner", "investigate"),
("prior delinquency", "all applications", 0.050, "quarterly", "model owner", "investigate"),
("mean utilisation", "all applications", 0.050, "quarterly", "model owner", "investigate"),
("discrimination", "matured 12 months", 0.050, "semi-annual", "model risk", "redevelop"),
("bad rate", "all applications", 0.050, "quarterly", "credit risk", "investigate"),
],
columns=["metric", "band", "level", "window", "owner", "action"],
)
# Every level is measured from the figures the model was approved on,
# quoted from the development sample of 4,000 applications.
baseline = {
"calibration gap": 0.000, # a gap is measured against zero
"thin-file share": 0.190,
"prior delinquency": 0.340,
"mean utilisation": 0.448,
"discrimination": 0.772, # refit without the leaked collections field
"bad rate": 0.292,
}
# Note what is NOT in this table: any row that would have caught the leak.
# Monitoring compares the model against the world, and the leak is a
# relationship the world keeps confirming. That one is session 2's job.# The later vintage: 2,000 applications from the following year.
observed = {
("calibration gap", "all applications"): -0.006, # predicted 0.330, actual 0.324
("calibration gap", "worst quintile"): -0.142, # quintile 5: 0.694 predicted, 0.552 actual
("thin-file share", "all applications"): 0.340,
("prior delinquency", "all applications"): 0.368,
("mean utilisation", "all applications"): 0.464,
("discrimination", "matured 12 months"): 0.682,
("bad rate", "all applications"): 0.324,
}
spec["observed"] = spec.apply(lambda r: observed[(r["metric"], r["band"])], axis=1)
spec["move"] = (spec["observed"] - spec["metric"].map(baseline)).abs().round(3)
spec["fires"] = spec["move"] >= spec["level"]
for _, r in spec[spec["fires"]].iterrows():
print(r["metric"], "/", r["band"], "moved", r["move"], "->", r["action"], "-", r["owner"])
# calibration gap / worst quintile moved 0.142 -> recalibrate - model owner
# thin-file share / all applications moved 0.15 -> investigate - model owner
# discrimination / matured 12 months moved 0.09 -> redevelop - model risk
print(spec.loc[~spec["fires"], ["metric", "move"]].to_string(index=False))
# calibration gap 0.006 <- the aggregate. It passes.
# prior delinquency 0.028
# mean utilisation 0.016
# bad rate 0.032
# Same model, same quarter, same pack: the aggregate calibration row passes
# and the row directly beneath it breaches by nearly three times the level.Run the spec against the period before the one you are worried about. If nothing fires on the quarter you already know went wrong, the spec is the problem and not the model. This is the cheapest check in the whole design and it is almost never done, because a control that has never fired is indistinguishable from a control that cannot.
Three questions
A monitoring pack reports one calibration number per model per quarter. On Brackwater's later vintage that number is -0.006. What is the problem with the design?
The aggregate is the average of the thing that matters, and averaging is the operation that destroys this particular signal. Under-prediction at the safe end and over-prediction at the risky end is the signature of a relationship that has moved, and it is made entirely of opposing errors. Putting the worst band on the threshold is what makes the row capable of going red for the right reason.
A monitoring row reads "thin-file share, alert if it deteriorates materially". What is missing before this counts as a control?
Four parts, and this row has none of them. Deciding the owner and the action at the moment of breach is exactly the failure mode: before go-live the level is an abstract question, and after a breach it is a question about taking a particular model off the road in a particular quarter, asked in front of the people who depend on it. A level set after a breach is a negotiation.
Suppose Brackwater had shipped the champion with the leaked collections field and monitored it for three years against a well-designed pack. What would the pack have shown?
Ongoing monitoring evaluates whether a model is performing as expected given changes in products, exposures, activities, clients, data relevance or market conditions. It is a comparison with the world, so it is structurally blind to a model that is wrong in a way the world confirms. Collections contacts keep being logged after default, the relationship keeps reproducing, and no threshold anywhere in the pack fires. Conceptual soundness is the only component that catches it, which is why it is the first one.
Before session 9
45 minTake the monitoring pack your team actually receives, and for every row in it write the four parts next to the number: the level, the window and population, the named owner, and the action on breach. Most rows will have one of the four and several will have none, and the count is the finding. Then do the harder half: for each row, name a specific way the model could go wrong and say whether that row would move if it happened. Any row where the honest answer is no should be marked for deletion, because it is currently being read as evidence of health. Bring the annotated pack to session 9, where what fires becomes a finding with a severity, a condition and an owner, and where the thing that gets argued about is no longer the number.
What tonight maps to
collections field would keep predicting default forever and no pack would ever fire. Conceptual soundness is the only component that catches it, which is why session 2 comes first.