Where this sits in the practitioner track
Sessions 3 to 7 produced evidence and session 8 turned two of those tests into standing controls. Tonight is the conversion that decides whether any of it mattered: evidence into findings, findings into severities, severities into conditions with owners and dates, and all of it into a register that outlives the meeting. Session 10 runs the whole workflow end to end against a model whose truth is known by construction, and it uses the shapes built tonight. The worked example is Brackwater's consumer scorecard, 4,000 development applications at a bad rate of 0.292 and 2,000 from the following year at 0.324, and by the end of tonight its four findings exist as rows somebody has to answer.
★ What you walk out with tonight. The five-part shape of a finding and a test that catches the part people leave out. A severity rule that survives being argued with, because it grades the consequence rather than the work. Brackwater's four findings written out in full at two severities, with the canon numbers attached to each. A register schema with ten columns, each of which exists because somebody later has to answer a question from it. And the honest version of the professional reality: findings get downgraded between the draft and the final, there are exactly two places it happens, and the defence is written evidence and a tracked register rather than a better argument on the day.
LiveFive parts, and the missing one is always the same4 min▶
What was observed. What evidence supports it. What the consequence is. What severity that consequence implies. What condition or remediation follows. Five parts, in that order, and the order matters because each one is answerable only once the one before it exists.
The part that goes missing is the third, every time. A validator who has spent two weeks finding something is full of the observation and the evidence, and writes both of them beautifully: here is the field, here is when it is written, here is what the model scores with and without it. Then the paragraph ends. The reader, who has a model to run and a committee on Thursday, is left to work out for themselves what goes wrong if nothing changes, and the answer they arrive at is usually "nothing much, for now".
Without a consequence there is nothing to grade, so there is no severity. Without a severity there is nothing to escalate, so there is no condition. Without a condition there is nobody's name on it and no date, so it does not enter the register, and a thing that is not in the register is a thing nobody will ever be asked about again. One missing sentence removes the entire chain.
Brackwater's earlier review did notice the collections field. The note in the file reads, in full, that the field is "predictive and available in the warehouse". Both statements are true, neither is a consequence, and the item was recorded as an observation. Two years and eight quarterly packs later nobody had ever been asked about it, because observations are not the kind of thing anybody is asked about.
Self-studyObservation, finding, condition, recommendation4 min▶
- Observation. Something true and noted, with no stated consequence. Perfectly legitimate as a category and dishonest as a landing place for something that has one. Observations are not tracked to closure, which is precisely why a contested finding gets pushed into this box.
- Finding. An observation plus a consequence plus a severity. It enters the register, it has an owner, and it has a status that has to reach a terminal value before anybody can say the review is closed.
- Condition. Something that has to happen for the model to be used, or to keep being used. It binds. It carries a name and a date, and failing it is itself an event with a consequence.
- Recommendation. Something the validator thinks would be an improvement, which the model owner may decline with a reason. Useful, and not a control. Mixing recommendations into the condition list is how condition lists stop being taken literally.
- Exception. A condition that has been formally not met, with an approval attached. It is a legitimate outcome, and the register has to be able to hold it, because the alternative is a condition that quietly stays open forever.
The reason to be strict about these five words is that the difference between them is the difference between something that happens and something that does not, and the words are the only place that difference is recorded.
Grade the consequence, not the work
13 min liveSeverity is where validation reports go wrong most often and most invisibly, because the wrong rule feels like the right one. The wrong rule grades effort and ingenuity: a finding that took two weeks of forensic work on a join condition feels like a big finding, and a finding you noticed in the first ten minutes feels small. The right rule ignores both. Severity is the size of what goes wrong if nothing changes, and the leak at Brackwater is high severity for one reason only, which is that it invalidates the number the approval was requested on.
LiveWriting each of the five so it cannot be softened5 min▶
Observed. One sentence, about the model or the process, phrased so that somebody could disagree with it precisely rather than generally. "The field collections counts contacts logged after the outcome window opens" is disagreeable precisely: a model owner who thinks it is wrong has to say when the field is written. "The feature selection process was not sufficiently rigorous" is disagreeable only generally, which means it is arguable forever.
Evidence. Whatever makes the observation checkable by somebody who was not there. A number with its method and its sample, a timestamp, a line in the model document with its page. The test is reproducibility rather than volume: three numbers somebody else can regenerate beat a page of narrative.
Consequence. What goes wrong if nothing changes, stated in the currency the reader spends. Not "performance is overstated" but "the performance in the model document does not describe a model that could have been run, so the approval rests on a number that was never available". Write it so that the reader cannot agree with the observation and still shrug.
Severity. A grade, from a scale defined before the review started, applied to the consequence you just wrote. If the scale is invented during the write-up it will be invented to fit whatever the political weather allows.
Condition. What has to happen, by when, owned by a name. And what follows if it does not, because a condition with no failure branch is a request.
LiveSeverity is about consequence, not about cleverness or effort4 min▶
The leak is the cleanest illustration available, because it scores low on every wrong measure. It took about ten seconds to find once somebody asked when the field is written. It required no technique, no unusual access and no statistics. The whole finding is the order of two events. By any effort-based standard it is a minor item, and a validator who graded by effort would file it below a hard-won piece of forensic work on an imputation rule.
It is high severity because of what it does. The approval for this model was requested on a development AUC of 0.863, and that number is partly produced by a field whose value is determined by the outcome. Refit honestly, the development figure is 0.772. The consequence is that the decision was taken on evidence that does not describe a model anybody could run, and everything downstream of that decision inherits the problem. Nothing about how easy it was to find changes any of that.
The rule runs the other way too, and that is the harder half to hold. A finding that took three weeks, that nobody else would have spotted, and that turns out to change the number in the fourth decimal place is a low-severity finding. Grading it higher because of the work is the single most common way a severity scale stops meaning anything, and once a scale stops meaning anything the high grades stop being believed, including the ones that are right.
Write the consequence before you pick the grade, and pick the grade only from the consequence. If the consequence sentence does not justify the grade on its own, without anybody knowing how the finding was made, then the grade is wrong or the consequence is badly written. Both are fixable and only one of them is a fix to the grade.
Self-studyConditions, remediation and what documentation is actually for4 min▶
A condition has three parts and each of them is load-bearing. An owner, who is a person: a team cannot be chased, a function cannot be emailed on a Friday, and "the modelling group" has never once completed a remediation. A trigger or a date: either a calendar date or an event that is certain to occur and easy to observe, such as the next credit committee or the next scheduled recalibration. And a consequence of its own, which is what happens if the date passes with nothing done. Two of the three is the common shape and it is the shape that quietly expires.
This is also where documentation earns its reputation, which in most organisations is poor. The current guidance is specific about what it is for: documentation helps maximise the likelihood of continuity of operations, including supporting the tracking of recommendations, responses and exceptions, and it can be used to more effectively help manage any model remediation efforts. Read that as a list of three jobs. Continuity, so the next person can run this. Tracking, so a recommendation, the response to it and any exception granted are all recoverable. Remediation management, so the work that was promised can be followed.
None of those three is served by a long document. All three are served by a register with stable identifiers, and a register is a small thing to maintain compared with what it prevents. The practical marker of a healthy one is that it can answer "what happened to that" for an item raised two years ago, in under a minute, without asking anybody who was in the room.
Four findings, written out
11 min liveEverything the track has produced comes down to four rows. Two are high severity and two are medium, and the grading is entirely a consequence of the consequences. Notice while reading them that no row says the model is bad and no row recommends the challenger. The defensible position throughout is narrower and much harder to argue with: the evidence submitted did not support the claim made, which is a statement about a document rather than a verdict on a model.
LiveThe two high findings5 min▶
The leaked field. Observed: the field collections counts collections contacts, which are logged after the outcome window opens, so its value is determined by the outcome the model predicts. Evidence: the model document reports a development AUC of 0.863; the field fitted alone on the same sample reaches 0.748; the champion refit without it reaches 0.772. Consequence: the performance in the model document does not describe a model that could have been run at the decision point, so the approval rests on a number that was never available. Severity: high. Condition: requote development and out-of-time performance without the field, and resubmit for approval on the restated numbers.
The band-level calibration error. Observed: predicted default rates are close to actual in aggregate and systematically wrong within every band. Evidence: out of time the model predicts a mean rate of 0.330 against an actual 0.324, an aggregate gap of -0.006, while by quintile of predicted risk the gaps run +0.061, +0.068, +0.035, -0.054 and -0.142. Consequence: the model under-prices the safest borrowers and over-prices the riskiest, and any use of the predicted probability as a level rather than as a rank is wrong in opposite directions at the two ends of the book. Severity: high. Condition: recalibrate before the score is used to set any price, and report calibration by band in every monitoring cycle rather than in aggregate.
Both are high for the same structural reason and it is worth naming: each one invalidates a specific use that the model has already been approved for. That is the thing a high grade is supposed to mean, and reserving it for that keeps the grade worth something.
LiveThe two medium findings, and why they are not high4 min▶
The benchmark result. Observed: no simpler alternative was built or reported during development, so the model document contains no evidence that the eleven-feature specification buys anything over a simple one. Evidence: out of time the honestly refit champion reaches 0.682 and a three-feature challenger reaches 0.676. Consequence: the organisation is carrying the build cost, the validation cost and the ongoing monitoring obligation of a complex model with no evidence on file that the complexity is earning them. Severity: medium. Condition: build and report a simple benchmark at the next scheduled revalidation, and include one in the development evidence for any future scorecard.
It is medium rather than high because nothing about it says the approved use is unsafe. The model still ranks. The finding is about an absent piece of evidence and an unjustified ongoing cost, which is real and is not the same as a decision taken on a number that does not exist. Note also what this finding must not say: it must not say the challenger is better. On this data the two are within noise of each other on discrimination and each wins on a different business axis, so the claim would be unsupported and it would hand the model owner an easy rebuttal that distracts from the point.
The population shift. Observed: the population scored by the model has moved away from the population it was developed on. Evidence: thin-file applicants rose from 19.0 percent to 34.0 percent between the vintages, prior delinquency from 34.0 to 36.8 percent, mean utilisation from 0.448 to 0.464, and the bad rate from 0.292 to 0.324. Consequence: the development testing in the model document is weaker evidence about the current book than it appears to be, concentrated exactly where the model has least evidence. Severity: medium. Condition: monitor the thin-file share against the approved baseline of 0.190 with the trigger set in the validation report, and bring forward the revalidation if it moves further.
Self-studyModel cards: one documentation format, and what it does not settle4 min▶
A model card, in the sense proposed by Mitchell and colleagues in 2019, is a short structured document accompanying a model. Its fields are intended use, out-of-scope use, training data, evaluation data, and performance disaggregated by group. It is one format among several that can carry model documentation, and it is named here because it is the format most people have heard of.
Two things it does well, and both are genuinely useful in a validation context. It forces intended use and out-of-scope use to be written down, as separate named fields that are conspicuously empty if nobody fills them. Using a model beyond its intended purpose introduces additional uncertainty and risk, so a format that makes the boundary an explicit field is doing real work. And it forces performance to be reported disaggregated rather than as a single headline, which is the same instinct that makes banded calibration the only version of that test worth running.
What it does not settle is everything else. A model card is a description, not a validation: nothing in it is independently checked, nothing in it constitutes effective challenge, and a card can be complete, well written and entirely consistent with a model that should never have been approved. Brackwater's champion would fill one out beautifully. Intended use, stated. Training data, described. Evaluation data, described. Performance, 0.863. Every field correct, every field sourced from the model document, and not one of them capable of surfacing that a feature is written after the outcome. The card would have recorded the wrong number accurately.
So treat it as a container. If the organisation wants its documentation in that shape, that is a reasonable choice and it satisfies the documentation expectation as well as any other layout. Just do not let the completeness of the card be read as evidence about the model, because completeness of a description and soundness of the thing described are unrelated properties.
The politics, stated plainly, because it is professional practice and not a rumour. Findings get downgraded between the draft and the final. It happens in exactly two places and both are easy to watch for. The first is the severity grade, where high becomes medium on the argument that the consequence is theoretical until it happens. The second is the word: a condition becomes an observation, and with that one substitution the item leaves the register, loses its owner and its date, and stops being something anybody will ever be asked about. Sessions 2 and 5 cover why the pressure exists and what makes challenge effective against it. The defence here is narrower and entirely mechanical: written evidence that reproduces, a severity scale that was fixed before the review started, a consequence sentence that stands on its own, and a register with stable identifiers where the original grade and every change to it remain visible. Arguing better on the day is not a defence. Having written it down before the day is.
Build the register, then put the four findings in it
18 min build-alongThe register is the artefact that outlives everything else in a validation. The memo gets read once, the meeting is remembered inaccurately, and the register is what somebody opens in two years to ask what happened to that. Ten columns, each present because a specific question gets asked of it later. Build the schema first, make it refuse the row people try to write, and only then put Brackwater's four findings through it.
Write the observation as a sentence somebody can disagree with precisely. Name the field, the process or the number. A sentence that can only be disagreed with generally will be, and the disagreement will outlast your involvement in the review.
Attach evidence that somebody else can regenerate. The number with its sample and method, the timestamp, the page of the model document. If a reader would have to take your word for any part of it, that part is not yet evidence.
State the consequence, or file the item as an observation and say so. If you genuinely cannot say what goes wrong when nothing changes, the honest move is the observation category, openly used. What is not honest is a finding-shaped paragraph with the third sentence missing, which reads as a finding and behaves as nothing.
Grade against a scale that existed before the review. Apply it to the consequence you just wrote, not to the work you just did. Ten seconds of insight can be high and three weeks of forensics can be low, and a scale that cannot produce both of those results is not being applied.
Write the condition with an owner, a date or trigger, and a failure branch. A person by name. A date, or an event certain to occur and easy to observe. And what happens if the date passes, because a condition with nothing on the other side of it is a request that expires quietly.
Give the item a stable identifier and put it in the register. Tracking recommendations, responses and exceptions is one of documentation's stated purposes, along with continuity of operations and managing remediation, and none of the three works without an identifier that survives the document being rewritten.
Record what you could not test as a limitation of the validation. The data you were not given, the code you could not run, the question that needed an owner who was unavailable. Even with sound modelling and rigorous validation, material model risk can remain, and a report that states its own blind spots is describing the residual risk rather than pretending it away.
Keep the draft, dated, alongside the final. Not as a weapon and not as a threat. As the thing that makes a downgrade visible as a decision somebody took, with a reason, on a date, rather than as a difference nobody can account for afterwards.
import pandas as pd
# Ten columns. Each one exists because somebody later has to answer
# a question from it, and none of them is decoration.
COLUMNS = [
"id", # stable, quoted in the memo and in every follow-up
"component", # conceptual soundness, outcomes analysis, ongoing monitoring
"observation", # one sentence, disagreeable precisely
"evidence", # what makes the observation checkable by somebody else
"consequence", # what goes wrong if nothing changes
"severity", # graded on the consequence, scale fixed before the review
"condition", # what has to happen
"owner", # a named person, never a team
"due", # a date, or an event certain to occur
"status", # open, responded, accepted, closed, exception
]
def finding(**kw):
"""Refuse to build a row that is missing the part people leave out."""
if not kw.get("consequence"):
raise ValueError("no consequence: this is an observation, file it as one")
if kw.get("severity") and not kw.get("condition"):
raise ValueError("graded but unconditioned: nothing would have to happen")
return {c: kw.get(c, "") for c in COLUMNS}register = pd.DataFrame([
finding(
id="MR-2026-014",
component="conceptual soundness",
observation="collections counts contacts logged after the outcome window "
"opens, so its value is determined by the outcome predicted",
evidence="submitted development 0.863; collections fitted alone 0.748; "
"champion refit without it 0.772",
consequence="the performance in the model document does not describe a "
"model that could have been run at the decision point",
severity="high",
condition="requote development and out-of-time performance without the field",
owner="model owner",
due="before the next credit committee",
status="open",
),
finding(
id="MR-2026-015",
component="outcomes analysis",
observation="calibration is close in aggregate and systematically wrong "
"within every band of predicted risk",
evidence="out of time, predicted 0.330 against actual 0.324, a gap of "
"-0.006, over quintile gaps of +0.061, +0.068, +0.035, "
"-0.054 and -0.142",
consequence="the safest borrowers are under-priced and the riskiest are "
"over-priced, so any use of the level rather than the rank is wrong",
severity="high",
condition="recalibrate before the score sets any price; report calibration "
"by band every cycle",
owner="model owner",
due="before the score is used in pricing",
status="open",
),
finding(
id="MR-2026-016",
component="conceptual soundness",
observation="no simple benchmark was built during development, so nothing "
"on file shows the eleven-feature specification buys anything",
evidence="out of time the honest champion reaches 0.682 and a three-feature "
"challenger reaches 0.676",
consequence="build, validation and monitoring cost is being carried for "
"complexity with no evidence on file that it earns its place",
severity="medium",
condition="build and report a simple benchmark at the next revalidation",
owner="model owner",
due="next scheduled revalidation",
status="open",
),
finding(
id="MR-2026-017",
component="ongoing monitoring",
observation="the population being scored has moved away from the population "
"the model was developed on",
evidence="thin-file share 19.0 to 34.0 percent; prior delinquency 34.0 to "
"36.8; mean utilisation 0.448 to 0.464; bad rate 0.292 to 0.324",
consequence="the development testing is weaker evidence about the current "
"book than it appears, concentrated where the model knows least",
severity="medium",
condition="monitor thin-file share against the approved baseline of 0.190 "
"with the trigger set in this report",
owner="model owner",
due="every quarterly monitoring cycle",
status="open",
),
])
print(register[["id", "component", "severity", "owner", "due", "status"]])
# Two high, two medium. The grades come from the consequence column and
# from nothing else: the leak took about ten seconds to find.When you cannot test something, that is a row too. Record it as a limitation of the validation, with the same discipline: what you could not test, why, and what risk therefore remains unmeasured. Even with sound modelling practices and rigorous validation, material model risk can remain, and the report that says where its own blind spots are is the one a supervisor can rely on. A validation that reads as complete when it was not is a worse document than one that names three things it could not reach.
Three questions
A validator writes: "The field collections is written after the outcome window opens. Development AUC falls from 0.863 to 0.772 when it is removed." What is missing, and why does it matter?
Observation and evidence are both there and both good. Without a stated consequence there is no severity, without a severity there is no condition, and without a condition the item never gets an owner or a date and is never tracked to closure. One missing sentence removes the whole chain, which is why it is the part that reliably goes missing.
The leak took about ten seconds to find once somebody asked when the field is written. A separate finding about an imputation rule took three weeks of forensic work and moves the reported number in the fourth decimal place. How should they be graded?
Severity grades the consequence and nothing else. The leak means the approval rests on a development figure of 0.863 that partly measures the outcome, against an honest 0.772. The imputation finding changes almost nothing. Grading by effort is the most common way a severity scale stops meaning anything, and once it does the high grades stop being believed, including the ones that are right.
A model owner proposes that the benchmark finding be recorded as an observation rather than a condition, since the model still ranks and nothing unsafe has been shown. What is the effect of that change?
Observations are not tracked to closure, and that is exactly what makes the box attractive for a contested item. The two places a finding gets softened between draft and final are the severity grade and the word "condition" becoming "observation". The defence is written evidence, a severity scale fixed before the review, and a register with stable identifiers where the original grade remains visible.
Before session 10
45 minFind the last validation report or review memo your organisation produced, and mark up every item in it against the five parts: observed, evidence, consequence, severity, condition. Count how many have all five and how many stop after two. Then take the items recorded as observations and ask, for each one, whether a consequence could have been written and was not. That count is the more interesting number. Finally, pick the single item you would most want to be able to find in two years and check whether it has a stable identifier, a named owner and a status that can reach a terminal value. Bring what you find to session 10, where the whole workflow runs end to end against a model whose truth is known by construction, and where the findings you write get checked against what is actually true.