What a validator is checking

The most common misreading of validation, from people who have never had one done to them, is that it is a quality-control pass on the computation. Somebody re-runs the numbers, confirms they come out the same, and signs. If that were the job it would be a week of work and it would catch almost nothing, because the modelling team has already run those numbers many more times than the validator will.

The real job is evidential. A model document makes a claim: this model separates the customers who will default from the ones who will not, this well, on this population, for this purpose. Validation asks whether the evidence attached to that document is good enough, and wide enough, to support the claim. Those are two separate tests. Evidence can be impeccable in quality and far too narrow in extent, which is exactly the shape of the Brackwater failure.

• Conceptual soundness. Assessing the model's design, its key modelling choices, its assumptions, the qualitative judgements built into it, how the data was selected and constructed, and what developmental testing was done. The test is on both the quality and the extent of the developmental evidence.
• Outcomes analysis. Comparing the model's outputs to the corresponding real-world outcomes. Back-testing and outlier analysis are named forms of it. Where a model rests substantially on expert judgement, quantitative outcomes analysis is how the quality of that judgement gets evaluated rather than simply accepted.
• Ongoing monitoring. Evaluating whether the model still performs as expected given changes in products, exposures, activities, clients, the relevance of the data, or market conditions.
• Benchmarking. Not one of the three. A comparison against an alternative, often a deliberately simpler one, that answers a question none of the three components asks on its own: was the complexity worth anything.
• Out of time. Scored on a later period than the one the model was built on. The single cheapest way to find out whether a result was a property of the model or a property of that year.
• Developmental evidence. What the builders produced while building: the tests they ran, the alternatives they tried, the choices they rejected and why. Its absence is itself a finding.

The numbers on this page are quoted, not explained. What a separation figure is, how calibration is computed and what a base rate does to both belong to learn-model-evaluation-with-phoebe, which defines every one of them properly.

Conceptual soundness is the part of validation that needs no new data and no waiting. It is an assessment of the model's design, the key modelling choices, the assumptions, the qualitative judgements, and how the data was selected and constructed, together with whatever developmental testing the builders did. The word to hold onto is extent. The review is not only whether the developmental evidence is any good; it is whether there is enough of it to support what is being claimed.

At Brackwater this is the component that found the leak. One of the eleven fields was a count of collections contacts. Collections contacts are logged after the outcome window opens, which means the field is a consequence of a customer defaulting rather than a signal that they are about to. On its own that single field separates defaulters from non-defaulters at 0.748, against the full model's 0.863. Refit without it, development performance drops to 0.772.

Notice what caught it. Not a performance test - the performance was excellent, which is precisely the problem. It was a question about what the field means and when it comes into existence, asked by somebody reading the data dictionary against the outcome definition. That question costs an afternoon and no data at all.

Outcomes analysis compares model output to the corresponding real-world outcomes. That is the whole idea, and everything else is a form of it: back-testing, where you score a period the model did not see and wait for the outcomes to arrive, and outlier analysis, where you go and look at the cases the model got most wrong rather than the average of all of them. The point worth carrying into a committee is the one about judgement. Where a model rests substantially on expert judgement, quantitative outcomes analysis is how the quality of that judgement is evaluated. Experience is not a substitute for a track record; it is a reason to go and measure one.

Ongoing monitoring asks whether the model still performs as expected given changes in products, exposures, activities, clients, the relevance of the data, and market conditions. It is the only component with no end date, and the only one that fails quietly, because nothing about a model going stale generates an alert unless somebody wrote the alert first.

At Brackwater it found the population. Between the development vintage and the later one, the share of thin-file applicants in the book went from 19.0 percent to 34.0 percent. Nobody changed the model and nobody made an error. The book moved toward exactly the customers the model had seen least of, and the only reason this is a finding rather than a surprise is that somebody was still looking a year after launch.

Conceptual soundness is the easiest component to fake, because it produces prose rather than tables and prose can be written without doing anything. Four tells, all of which you can check in a report without any technical background.

• Are the assumptions listed as assumptions? A section that describes the methodology is not the same as a section that names what is being presumed true. If the word "assumes" appears nowhere, nobody wrote them down.
• Is any alternative mentioned? Developmental evidence includes what was tried and rejected. A document in which the chosen approach is the only approach ever considered is describing a decision that was never made.
• Does every field have a provenance and a timing? Where it comes from and when in the customer's life it gets written. This one question is what caught Brackwater's leak, and it is answerable by anybody.
• Is the extent of the evidence assessed, or only its quality? "The tests performed were correct" and "the tests performed were sufficient" are different sentences, and reports routinely contain only the first.

The mechanism is simple enough to say in a sentence. The model is too optimistic about the customers it thinks are safe and too pessimistic about the customers it thinks are risky. In the first three bands the actual default rate comes in above what was predicted, by +0.061, +0.068 and +0.035. In the last two it comes in below, by -0.054 and -0.142. Add them all up across the whole book and they very nearly wipe each other out, leaving the -0.006 that looks like a triumph.

Both halves of that cost money, in different places and to different people. Under-predicting the safest band means the customers being approved most freely, at the best prices, default more often than the pricing assumed. Over-predicting the riskiest band means declining or repricing customers who would mostly have paid. One error shows up in losses and the other shows up in volume that never happened, which is why only the first tends to get noticed.

The fourth finding came from a comparison rather than from any of the three components. The validator built a challenger: a deliberately plain model using three features, the kind of thing a competent analyst produces in an afternoon. Scored on the later vintage, the champion separates defaulters from non-defaulters at 0.682, with the leaking field removed. The challenger scores 0.676. The difference is 0.006.

Be careful with what that does and does not establish. It does not establish that the champion is worthless or that the challenger should be deployed instead; on this data the two are close enough that the ranking is not the point, and each has different consequences for volume and for losses. What it establishes is the thing a leader needs: the model document claimed an edge, and the edge that survives to the population the model is actually being used on is six thousandths over something nobody needed a project for. The defensible finding is not about the model. It is about the document, which asserted more than its evidence supported.

The guidance is unusually direct about its own limits, and it is worth reading that part rather than skipping to the procedures. Even with sound model development and rigorous validation, material model risk can remain. Validation reduces risk; it does not retire it, and a model that has passed is not a model that has been made safe.

What follows from that is a short list of habits for the people who use the output, and none of them are the validator's job. Understand the model's limitations and communicate them, rather than passing the number on stripped of its caveats. Monitor performance instead of assuming the launch conditions persist. Review periodically whether the model is still relevant to the decision it is feeding, which is a different question from whether it still performs. And supplement the model's output with complementary analysis, so that no consequential decision rests on one number from one source.

That last habit is the one leaders can actually enforce, and it is worth noticing that it is the same discipline as the band question. One number, from one model, read on its own, is the failure mode that runs through every finding on this page.