What the bench measures
Whether your findings are right. Not whether they sound rigorous, not whether the report is well written, but whether the thing you wrote down as a finding is actually true of the model in front of you. That is a check no real validation ever gets, because in a real validation nobody knows the answer. Here the applications come from a model written out in the source file, so every claim the bench makes can be traced to a line of it.
★ What you walk out with tonight. A worked validation of one model, start to finish, with the evidence for each finding computed rather than asserted. The measured answer to the question every model owner asks - what would the simpler thing have scored - which here is 0.676 against 0.682. And the habit the whole track has been building toward: writing the recommendation the evidence supports, which on this model is neither approval nor refusal.
LiveWhere a known answer comes from3 min▶
Real applications never come labelled with the reason they defaulted. These do, because they were generated from a written model: a borrower's true default probability is a stated function of utilisation, prior delinquency, months employed, income ratio and whether the file is thin, plus a shock nobody has a column for. Two vintages come out of it. The development sample is 4,000 applications with a bad rate of 0.292; the later vintage is 2,000 with a bad rate of 0.324.
Three things move between the two, all deliberately: the share of thin-file applicants, the strength of the utilisation relationship, and the size of the unobserved shock. That is what makes this a validation exercise rather than a modelling one. Nothing about the champion is broken. The world it was fitted to stopped being the world it runs in.
Brackwater's review process was a model document, a meeting, and a sign-off sheet. Every number in the document was correct. The reason the review passed is that nobody asked what the numbers were evidence of, and the one question that would have unravelled it - when is the collections field written - takes about ten seconds to ask and was never asked in two years of quarterly reviews.
Self-studyWhat is real here, and what is not3 min▶
- The data. 6,000 applications from fixed seeds, identical for every learner, generated by the model written out in
mr-live.jswhere every coefficient is visible. - Every model. A real logistic regression, fitted in the page by iteratively reweighted least squares. The champion, the refit and the challenger are all estimated when you click, not stored.
- Every AUC. A rank-based calculation over the scores those regressions produced. Nothing is hard-coded, and a number that disagreed with the course notes would be the course notes' problem.
- The leak. Real in the strict sense: the collections field is generated from the default outcome, so its predictive power is a fact about the data-generating process rather than a claim about the world.
- What is not modelled. There is no simulated human behaviour anywhere in this widget, and no finding is written in advance. The findings panel is assembled from what the tests return.
Seven tests, three components, one decision
10 min liveThe tests on the bench are not a checklist somebody invented. Each one belongs to one of the three components of validation, and part of the exercise is being able to say which, because a report organised by component is a report a reviewer can follow and a report organised by whatever you happened to run is not.
LiveThe number that decides the report4 min▶
Of the seven tests, the benchmark is the one that changes the recommendation rather than the wording. The champion, honestly refit and scored on the later vintage, reaches 0.682. A challenger using prior delinquency, utilisation and months employed reaches 0.676. Eleven features, a bespoke build, a validation cycle and an ongoing monitoring obligation, for six thousandths of AUC.
That is not an argument for the challenger. It is an argument about evidence: nothing in the model document ever established that the complexity was buying anything, because nobody had run the simple thing. A benchmark is cheap and it reframes every other finding, which is why it belongs early in a validation rather than at the end.
Self-studyWhy the aggregate calibration number is the dangerous one3 min▶
Scored on the later vintage the model predicts a mean default rate of 0.330 against an actual 0.324. A gap of 0.006 on the headline would pass any monitoring threshold anybody writes. Split it by quintile of predicted risk and the same model under-predicts the safest band by 0.061 and over-predicts the riskiest by 0.142.
The errors are large, they are systematic, and they cancel. A monitoring pack that reports one calibration number per model per quarter would show this model as healthy for as long as the two halves stay in balance, and would show it as suddenly broken on the quarter they stop. Banding is not extra credit; it is the only version of the test that can fail for the right reason.
Run the validation
26 min build-alongWork left to right. Run a test, read what it returns, and watch the findings panel assemble itself. Then make the call. The bench is not scoring your speed and there is no prize for finding everything: the point is the gap between what you would have written after three tests and what the evidence supports after seven.
Run them in a deliberate order at least once. Start with the feature list rather than the metrics. A validator who scores the model first spends the rest of the review explaining a number that should never have been quoted, and a validator who reads the data dictionary first finds the leak before the number has a chance to anchor anybody.
Turn seven results into one memo
6 min liveThe bench produces findings. A report is findings plus a recommendation plus the conditions that make the recommendation safe, and the conditions are the part that gets written badly, because a condition with no owner and no date is a sentence rather than a control.
State the claim you are validating, not the model. "The scorecard ranks default risk well enough to set the approval cutoff for the 2026 book." Everything that follows is evidence for or against that sentence.
Lead with the finding that changes the decision. Here that is the leaked field, because it invalidates the number the approval was requested on. Severity is about consequence, not about how clever the finding was.
Requote performance honestly. 0.682 out of time, with the method stated, replacing the 0.863 in the model document. A validation that removes a number without supplying its replacement gets ignored.
Give each condition an owner and a trigger. Recalibrate before the score drives pricing. Monitor the thin-file share with a stated threshold. Re-run the benchmark at the next review. Each one names a person and a date or it is decoration.
Say what you did not test. The limitations of the validation are part of the validation. Even with sound modelling and rigorous review, material model risk can remain, and the report that admits its own blind spots is the one a supervisor trusts.
Three questions
On the later vintage the model predicts a mean default rate of 0.330 against an actual 0.324. What has that told you?
A single calibration number would have passed this model. The bands are what make the test capable of failing for the right reason, and the aggregate gap of -0.006 is the number that hides them.
Out of time the champion scores 0.682 and a three-feature challenger scores 0.676. What is the defensible finding?
The two are close enough that neither discrimination result separates them, and each wins on a different business axis: at matched cutoffs the champion approves 69.3 percent with 24.7 percent bad among approved, the challenger 72.2 percent with 25.3 percent. The finding is about the evidence submitted, not about which model is better.
You have run every test and found one high-severity leak, systematic band-level calibration error, and a population that has moved. What does the evidence support?
Refusal is for a model that cannot be used safely at all. This one can be used on a narrower mandate with its number restated and its drift watched, which is what approval with conditions means when the conditions have owners and triggers attached.
After the track
45 minTake a model your team owns and run the two cheapest tests from tonight against it: score every feature on its own and see whether any single field approaches the full model, and build the three-column benchmark. Both fit in an afternoon and neither needs anybody's permission. Write down what you find in the form the bench uses - finding, severity, condition - and notice how much of a validation report you can produce before anybody has been asked for a meeting.