Where this sits in the practitioner track
Session 3 found the leaked field and produced the honest development number, 0.772 against the submitted 0.863. Session 4 reproduced the document and showed what one unrecorded choice is worth. Tonight supplies the context both of those findings were missing: a floor. Without one, a validator reporting a 0.091 leak has no way to say whether the model underneath it was worth building. Session 6 then takes the honest model and asks a different question, whether its predictions match what actually happened. Brackwater stays the worked example, 4,000 development applications at a bad rate of 0.292 and 2,000 from the following year at 0.324.
★ What you walk out with tonight. A challenger you can build in twenty lines and defend in one sentence, and the rule for choosing one: the simplest thing the business could actually run, on data that definitely exists at scoring time. The Brackwater comparison on both axes, discrimination and business impact, where the champion wins one and the challenger wins the other. The claim not to make, which is that either model is better. And the claim that is defensible and survives contact with a committee: the evidence submitted did not support the claim made, which is a statement about the model document rather than about the model.
LiveWhat a benchmark is for4 min▶
A model document reports a number. Every number needs a comparison before it means anything, and the comparison a model document supplies is almost always the wrong one: last year's model, or a target agreed at the start of the project, or nothing at all. The comparison a validator needs is the one nobody has an interest in producing, which is what the same business outcome would have cost with a model that took an afternoon.
That is the entire idea of a benchmark. It is not an attempt to build a better model and it is not a competition. It sets a floor under the reported number so that the rest of the review has a scale to work on. A validation that reports performance without a floor is reporting a number in units nobody can interpret, and the interpretation gets supplied later by whoever is most confident in the room.
Brackwater's model document contains a comparison. It compares the champion's 0.863 against the previous scorecard's number from four years earlier, on a different book, computed by a team that had since left. It is the only benchmark in the file, and it was the strongest evidence in the approval pack. Nobody had ever fitted three columns on the current sample, which takes about twenty lines and would have taken an afternoon at any point in the model's life.
Self-studyFour words for four different comparisons3 min▶
- Benchmark. A reference model run on the same sample to put the reported number on a scale. Its job is to be interpretable, not to be good. It is never proposed for deployment.
- Challenger. A model kept alongside the champion, scored on the same applications over time, and capable of replacing it. A challenger is an operational commitment; a benchmark is a one-off calculation. Tonight's three-feature model starts life as the first and could become the second.
- Baseline. The trivial reference: predict the base rate for everybody, or keep whatever rule the business used before there was a model. Worth computing because it is free and because it occasionally wins.
- The incumbent. Whatever is running today. Useful to compare against and easy to over-weight, because the incumbent was fitted on a different book and its reported number was produced under a different standard of evidence.
Keeping these apart matters in the write-up. A recommendation to run a challenger programme commits somebody to monthly scoring and a governance process. A benchmark commits nobody to anything, which is precisely why it is the one a validator can always insist on.
Three columns, and a gap of six thousandths
11 min liveThe challenger uses prior delinquency, utilisation and months employed. All three are known at the moment the application is decided, all three are in every credit file, and any analyst at Brackwater could have produced the model in an afternoon. It scores 0.763 on development and 0.676 on the later vintage. The champion, refit without the leaked field, scores 0.772 and 0.682. The champion's edge out of time is 0.006.
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
dev = pd.read_csv("brackwater_dev.csv") # 4,000 applications, bad rate 0.292
oot = pd.read_csv("brackwater_oot.csv") # 2,000 from the following year, bad rate 0.324
# Three fields, all of them known at the moment the application is decided.
challenger = ["priorDelq", "util", "employed"]
# The champion, honestly refit: the leaked field from session 3 is gone.
no_leak = ["util", "priorDelq", "employed", "incomeRat", "thin",
"age", "n0", "n1", "n2", "n3"]
def fit(cols):
return LogisticRegression(max_iter=2000, C=1e4).fit(dev[cols], dev["bad"])
def auc(model, cols, frame):
return roc_auc_score(frame["bad"], model.predict_proba(frame[cols])[:, 1])
ch, cp = fit(challenger), fit(no_leak)
print(round(auc(ch, challenger, dev), 3)) # 0.763 on development
print(round(auc(ch, challenger, oot), 3)) # 0.676 on the later vintage
print(round(auc(cp, no_leak, dev), 3)) # 0.772
print(round(auc(cp, no_leak, oot), 3)) # 0.682
# Eleven features against three, out of time: an edge of 0.006.LiveWhy the benchmark belongs early in the review4 min▶
A benchmark is usually run last, if at all, on the grounds that it is a nice-to-have once the real tests are done. That ordering wastes it, because the benchmark is the finding that reframes the others rather than adding to them. Consider session 3's leak, worth 0.091 on development. Read it against a model that beats three columns by 0.15 and it is a serious defect in a genuinely valuable model, and the report is about fixing the number. Read it against a model that beats three columns by 0.006 and the same leak becomes the whole story, because the leak was the only thing holding the complexity up.
Identical finding, identical size, two different reports. The only thing that changed is whether there was a floor on the page. Run the benchmark before you decide how to frame the leak, the calibration work or the monitoring recommendation, and all three come out at the right weight the first time.
There is a cost argument too, and committees respond to it. Eleven features carry eleven data dependencies, eleven things that can drift, and eleven lines of an ongoing monitoring obligation. Three carry three. That comparison is not part of the AUC and it belongs in the report next to it.
LiveHow to choose a challenger4 min▶
The rule is one sentence: the simplest defensible thing the business could actually run, built on data that definitely exists at scoring time. Every word in it is doing work. Simplest, because a complicated challenger gives the model owner somewhere to argue. Defensible, because a challenger that a credit officer would laugh at proves nothing when it does well. Could actually run, because the comparison is about what the business gave up, not about what is theoretically possible. Data that definitely exists at scoring time, because a challenger built on a post-outcome field repeats session 3's mistake with the validator's name on it.
Brackwater's three fields pass all four. Prior delinquency, utilisation and months employed are in the credit file at application, they are the fields a human underwriter would have asked for, and nothing about them needs explaining to a committee.
A challenger nobody could deploy is not a benchmark, it is a debating point. If your reference model needs a data source the business does not have, or a field that arrives two weeks after the decision, it cannot tell you what the complexity was worth. It can only tell you what a different business would have scored.
Self-studyWhat the benchmark does not license3 min▶
It does not license a recommendation to deploy the challenger. Replacing a production scorecard is a modelling and business decision with its own build, its own validation and its own implementation risk. The validator's finding is about the evidence for the champion, and a validator who arrives with a replacement has stopped being a reviewer.
It does not license a claim about which model will do better next year. Both numbers come from one later vintage. What holds up over time is a monitoring question, and a challenger run alongside the champion for a few cycles answers it far better than a single comparison can.
It does not license dropping features. Session 3 showed that four of the champion's features are pure noise and that ten of the eleven carry very little. Tonight shows that the whole set beats three columns by 0.006. Neither is a simplification instruction. Both are evidence that the design was never justified, which is a finding the developer answers with a redesign or with an argument, not something the report decides for them.
What it does license is a condition: re-run the benchmark at every periodic review, alongside the performance numbers, so the question is never again unanswered for the life of the model.
Where it belongs, and what it costs the book
10 min liveTwo things have to happen before a benchmark result is usable. It has to be filed in the right place, and the guidance is specific about that: benchmarking to other models is named inside conceptual soundness, as one practical way to assess a model, and not as a component of its own. Then it has to be translated, because a committee does not act on 0.006. Set the cutoff on the development sample at the 75th percentile of score, apply it to the later vintage, and count. The champion approves 69.3 percent of the later vintage with 24.7 percent bad among approved. The challenger, at its own equivalent cutoff, approves 72.2 percent with 25.3 percent bad. The champion buys 0.6 points of bad rate and gives up 2.9 points of volume.
import numpy as np
def business(model, cols):
"""Set the cutoff on development, then read the later vintage through it."""
cut = np.quantile(model.predict_proba(dev[cols])[:, 1], 0.75) # safest 75 percent
s = model.predict_proba(oot[cols])[:, 1]
ok = s <= cut
return round(100 * ok.mean(), 1), round(100 * oot["bad"][ok].mean(), 1)
print(business(cp, no_leak)) # (69.3, 24.7) champion
print(business(ch, challenger)) # (72.2, 25.3) challenger, equivalent cutoff
# 0.6 points of bad rate bought for 2.9 points of volume given up.
# Which of those the business would rather have is not the validator's call.
# Reporting that the trade exists, and that nobody had measured it, is.LiveThe claim not to make4 min▶
Do not write that the champion is worthless, and do not write that the challenger is better. Both sentences overreach the evidence in front of you. On this data the two models are within noise of each other on discrimination, and on business impact each one wins a different axis: the champion takes fewer bad accounts, the challenger takes more volume. There is no reading of these numbers that ranks one above the other without also importing a business judgement about the price of volume, and that judgement is not the validator's to make.
What is defensible, and what the report should say, is that the evidence submitted did not support the claim made. The model document claimed a level of performance that justified an eleven-feature bespoke build and the ongoing obligations that come with it. Nobody had ever established that the complexity bought anything, because nobody had run the comparison. That sentence is about the model document. It does not require you to prefer either model and it cannot be rebutted by preferring one.
The test for whether your finding overreaches. Read it back and ask which sentence the model owner could argue with using a business fact you do not have. "The challenger is better" invites exactly that argument and you will lose it. "The comparison was never run, and here is what it shows" invites no argument at all, because it is a fact about the submission.
LiveWhy matched cutoffs are the honest comparison3 min▶
Two models produce scores on different scales, so comparing them at the same numerical threshold compares nothing. The fix is to give each model the cutoff that implements the same business policy, which here is approving the safest 75 percent of the development sample, and then to apply each model's own cutoff to the later vintage. Both models are then being asked to carry out the same instruction, and the differences that appear are differences in how well they carry it out.
Notice what the champion's own numbers do across the two vintages under that policy. It was set to approve 75.0 percent on development with 19.5 percent bad among approved. On the later vintage the same cutoff approves 69.3 percent with 24.7 percent bad. The book got harder for both models, which is a session 7 subject, and it is worth flagging here so nobody reads the 69.3 as a property of the cutoff rule.
One caution on the arithmetic. These are two figures from one later vintage of 2,000 applications, and a difference of 0.6 points of bad rate on a sample that size is not something to build a policy on. Report it as what it is: the scale of the trade, measured once, on the data available.
Self-studyWhere benchmarking sits in the guidance3 min▶
Validation has three components under the revised interagency guidance of 17 April 2026: conceptual soundness, outcomes analysis and ongoing monitoring, in that order. Benchmarking is named inside the first of those. Assessing conceptual soundness means assessing model design, construction and developmental testing; for some models what matters is evaluating the theoretical construction, and for others, in the guidance's own framing, interpretability measures or benchmarking to other models are more practical.
So benchmarking is one practical way to assess a model, sitting inside conceptual soundness. It is not a fourth component. This is worth saying plainly because the opposite is drawn constantly, in slide decks and in validation standards, as a fourth box in a row of boxes. Anyone doing that is making an editorial claim about how a validation should be organised, which is a reasonable thing to argue for and a different thing from attributing it to the guidance.
The practical consequence is about where the finding goes in the report. A benchmark result is evidence about whether the model's design and construction were justified, which is conceptual soundness. Filing it under performance turns it into a debate about which number is bigger, and the numbers here are six thousandths apart.
Build the floor, then price the gap
18 min build-alongThe two code blocks above are the whole exercise: one fits the challenger, one turns the comparison into approvals and losses. The steps below are the order to run them in and the decisions to make while you do, because the choices that decide whether a benchmark is defensible are all made before any code is written.
Choose the three fields before you look at any performance number. The simplest defensible thing the business could run, on data that definitely exists at scoring time. Choosing them after seeing which ones score well turns the benchmark into a second model built by you, and it will not survive the first question about it.
Fit the challenger on the same sample as the champion, with the same procedure. Same rows, same fitting settings. Any difference between the two fits that is not the feature list is a difference you will have to explain later.
Compare on the later vintage, not on development. Development says 0.772 against 0.763 and out of time says 0.682 against 0.676. The second pair is the one the business will live with, and it is the one that closes the gap.
Set each model's cutoff by policy, not by score. The safest 75 percent of the development sample for both, then carry each cutoff forward. Comparing two score scales at the same numerical threshold compares nothing.
Report both axes together, in the same table. 0.682 against 0.676 on discrimination; 69.3 percent approved with 24.7 percent bad against 72.2 percent with 25.3 percent. Reporting either axis alone produces a ranking that the other axis contradicts.
Write the finding about the document, not about the models. The evidence submitted did not support the claim made. Then attach the condition: re-run this comparison at every periodic review, named owner, stated date.
Twenty lines and an afternoon is the number to quote. When a committee asks why the comparison was not part of the original submission, the answer that lands is the cost of producing it. Three fields, one fit, one table. The reason it was missing is not that it was expensive.
Three questions
Out of time the champion scores 0.682 and the three-feature challenger scores 0.676. What goes in the report?
The two models are within noise of each other on discrimination and each wins a different business axis, so neither A nor C can be defended from these numbers. The finding is about the submission: the comparison had never been run, and running it shows the edge is 0.006. That sentence describes the model document and cannot be rebutted by preferring either model.
With the cutoff set on development at the 75th percentile of score, the champion approves 69.3 percent of the later vintage with 24.7 percent bad among approved; the challenger approves 72.2 percent with 25.3 percent. What does this establish?
Both models are carrying out the same policy through their own cutoffs, and the result is a trade rather than a ranking: fewer bad accounts against less volume. Whether 0.6 points of bad rate is worth 2.9 points of volume is a pricing question for the business. What the validator reports is that the trade exists, its size, and the fact that nobody had measured it before the model was approved.
Under the revised interagency guidance of 17 April 2026, where does benchmarking sit?
Validation has three components in the current text, in the order conceptual soundness, outcomes analysis, ongoing monitoring, and benchmarking to other models is named inside the first as a practical alternative to evaluating theoretical construction. Drawing it as a fourth box is an editorial claim about how to organise a validation rather than something the guidance says. Note that SR 11-7 is superseded.
Before session 6
40 minPick a model your team owns and name its three-field challenger before you fit anything: the simplest defensible fields, all available at scoring time, written down with your reasons. Then fit it on the same sample the model was built on, score both on the most recent data you have, and put the two numbers side by side. Add the business translation if you can get the cutoff the model actually runs at. Whatever the gap turns out to be, write the one-sentence version of the finding and notice how much of it is about the model document rather than the model. Session 6 takes the honest champion and stops asking how well it ranks, asking instead whether its predicted rates match what happened.