Where this sits in the practitioner track
Session 1 built the inventory and made the tiering call. Session 2 read Brackwater's design before touching a number. Session 3 found the leaked field by reading the data dictionary, and produced a second number, 0.772, from the same data. Tonight goes back to the first number and asks the question nobody asks about it: can it be regenerated. Session 5 then asks what the simple thing would have scored. The worked example stays Brackwater's consumer scorecard, 4,000 development applications at a bad rate of 0.292, with a later vintage of 2,000 at 0.324.
★ What you walk out with tonight. The five things a model document has to state before a re-run is even possible, which doubles as a checklist you can run against a document before you have any data. The four explanations for a replication gap, in the order they actually occur, none of which is fraud and all of which invalidate the number. A worked re-fit of Brackwater's champion that lands on 0.863, and a second fit that changes one undocumented thing and lands 0.090 lower. And the wording for both outcomes, because "we reproduced the reported figure" is a finding worth writing down and so is "we could not, and here is what was missing".
LiveReplication is not about arithmetic4 min▶
Replication is the cheapest test on the bench and the one most often skipped, and the reason it gets skipped is that it feels like doubting somebody's arithmetic. It is not that. Nobody in a model review suspects the developer of dividing wrong. The question replication answers is different and much more useful: can the documented result be regenerated from the documented inputs, by a person who was not in the room, working only from what is written down.
Phrased that way it stops being an insult and starts being a property of the document. A number that only exists in one notebook on one laptop is not evidence, however correct it is. A number that a second person can produce from the written specification is evidence, because the specification is now known to be complete enough to describe the model that exists. That is the real output of a replication: not a verdict on the number, a verdict on the document.
Brackwater's champion reproduces on the first attempt. The feature list in the document is the feature list that was fitted, the sample is the sample, and the development AUC comes back as 0.863 to three decimal places. Two years of quarterly reviews had never checked, on the reasonable-sounding grounds that the number had been reviewed before. It had been read before. It had never been regenerated.
Self-studyThree things replication is not3 min▶
- Not a code review. You are not reading the developer's script for defects. You are writing your own from the specification and seeing whether it lands in the same place. If you start from their code you have tested nothing, because their code carries their undocumented choices with it.
- Not a performance test. A replication that reproduces 0.863 has said nothing about whether 0.863 is a good number, whether it survives a later vintage, or whether it was earned by a field written after the outcome. Those are other sessions. This one asks only whether the document regenerates.
- Not a fraud check. Every common cause of a gap is an omission rather than a misstatement, and treating a gap as an accusation is the fastest way to lose the cooperation you need to close it. The finding is that something was not written down.
What it is, and the reason it goes early in a validation: a cheap test of whether the rest of the review has a stable foundation. Every later finding is expressed relative to a reported number. If that number cannot be reproduced, every later finding is expressed relative to something nobody can regenerate.
Five things, and the two ways a re-run can end
11 min liveA validator can decide whether a model is replicable before running anything, by reading the document for five specific items. Most model documents carry three of them. The two that go missing are the ones nobody thinks of as part of the method: the exact sample boundary, and the population the metric was computed over. Both are invisible in the notebook, because in the notebook they were just whatever was in memory at the time.
LiveA pass is worth writing down4 min▶
Validators under-report passing tests, and replication is the one they under-report most, because a test that found nothing feels like an hour with no output. It has an output. It establishes that the model in production is the model in the document, which is a statement the approver is relying on whether or not anybody has checked it. Once it is checked, every other finding in the report inherits a fixed reference point: when the report says the honest number is 0.772 rather than 0.863, both numbers now refer to the same fitted object.
There is a second reason, which is about the next validator rather than this one. A replication that passed and was recorded, with the specification it was run from, turns the model document into something a successor can pick up. A replication that passed and was not recorded leaves the next person to spend the same hour, and they will not spend it, because the file gives them no reason to think it is needed.
The sentence to write. "The reported development AUC of 0.863 was independently reproduced from the documented sample, feature list and fitting procedure." One line, in the evidence section, with the date. That is the whole deliverable of a passing replication and it is worth more than its length suggests.
LiveWhat the guidance asks of developmental evidence4 min▶
Under the revised interagency guidance of 17 April 2026, validating conceptual soundness involves assessing and documenting model design, including key modelling choices, assumptions, qualitative judgements and data selection, together with construction and developmental testing. The standard of assessment is stated plainly: sound practice subjects modelling aspects to critical analysis by evaluating both the quality and the extent of the developmental evidence.
Those two words are what replication serves. Quality asks whether the evidence supports the claim. Extent asks how much evidence there is and whether it covers what it needs to. A validator who reads the developmental testing section and nods has assessed neither. A validator who re-runs it has tested both at once, because a specification too thin to re-run is by definition a specification whose extent cannot be assessed, and a result that does not regenerate is a result whose quality cannot be assessed either.
Note the vintage. SR 11-7, issued in 2011, is the document most of the industry still quotes from memory, and it was superseded. The current text is the April 2026 revision, issued jointly by the Federal Reserve, the OCC and the FDIC.
Self-studyThe two items documents usually miss3 min▶
The sample boundary. Documents say "the 2024 book" or "twelve months of applications". Neither is a sample definition. A sample definition is a rule that a second person can apply to the warehouse and get the same row count: which table, which date field, which inclusive boundaries, and which exclusions, in the order they were applied. Exclusions are where the row count goes: declined applications, staff accounts, records with a missing outcome, duplicates on a customer key. Each one is defensible and each one moves the sample.
The population the metric covers. A metric is a formula and a set of rows, and documents record the formula. If the reported figure was computed on the fitting sample, say so. If it was computed on a holdout, say which one and how it was drawn. If rows with a missing outcome were dropped before scoring but after sampling, that is a third population again. The formula is never the ambiguous part.
Both items share a property that makes them easy to lose: in the notebook they were not decisions, they were just the state of a variable. Nobody omits them on purpose. They were never in anybody's head as things to write.
Four explanations, in the order they actually occur
10 min liveWhen a re-run disagrees with the document, the useful move is not to escalate and not to assume error. It is to work down a short list in order of likelihood, because the list is short, the causes are ordinary, and the one you land on determines what the finding says. None of the four is fraud. All four mean the reported number cannot be relied on until the gap is closed, which is an uncomfortable pair of facts to hold at once and the reason this finding gets softened in the write-up.
LiveWhy the order matters more than the list4 min▶
Anybody can produce the list. The order is the working technique, because each item is cheaper to test than the one below it and each one, if it is the cause, makes testing the rest pointless. Start by comparing row counts and column names, which takes a minute and catches the first item outright. If the sample sizes disagree, nothing about preprocessing, seeds or metric populations is worth investigating until the sample question is settled.
Only when the sample matches does preprocessing become the live hypothesis, and the test for it is to ask for summary statistics on each feature rather than the code: minimum, maximum, mean, missing count. A capped variable shows up immediately in a maximum that is suspiciously round, and a filled variable shows up in a missing count of zero where the raw table has thousands.
The seed comes third because its signature is distinctive: a gap that changes size every time you re-run. If your re-run gives a different answer each time, you have found the cause without needing the developer at all, and the finding writes itself. The metric population comes last because it is the hardest to test from outside and, in practice, the rarest of the four in a scorecard.
LiveHow big a gap has to be before it is a finding3 min▶
This is the question the room always asks, and the honest answer is that the size of the gap is the wrong axis. A gap of any size is evidence that the document is incomplete, because a complete specification run twice gives the same answer twice. The size tells you about severity and about how hard you should push, not about whether there is something to report.
The practical split. A gap you can explain, where the developer names the missing step and the re-run then matches, is a documentation finding: the specification was incomplete, here is what was missing, add it. A gap you cannot explain is a different animal, because now neither party can say what produced the reported number, and a number nobody can account for should not be carried forward into an approval whatever its size.
What the size does drive is the rest of the review. A gap of three thousandths changes no conclusion and gets a documentation condition. A gap large enough to move the model across a threshold, a tier boundary or a business case has to be resolved before anything downstream of it is worth computing.
Self-studyAsking for what you need without asking for the code3 min▶
The request that destroys a replication is "can you send me your notebook". It feels cooperative and it ends the test, because from that moment you are re-running their choices rather than the written ones, and the thing you were measuring was exactly the difference between those two sets.
The requests that preserve it are narrow and answerable. The sample definition as a query or as a rule, not as a row count. Per-feature summary statistics on the fitting sample, which let you verify your sample against theirs without either party sharing code. The split rule, or the seed, stated as a value. And the population the reported metric was computed over, in rows.
If the answer to any of these is "it is in the code", that is itself the finding, and it is worth stating in exactly those terms: the specification for this model is the code, so the model cannot be independently reproduced from its documentation. That sentence is more useful to an approver than any number in the report, because it describes a condition that will still be true for the next model the same team builds.
Re-run the champion, then change one undocumented thing
18 min build-alongTwo fits. The first rebuilds Brackwater's champion from the documented specification and lands on the documented number. The second keeps the specification identical and changes the one thing the document does not record, the window the sample was drawn from, and watches the number move by 0.090 without a line of the model changing. That second fit is the whole argument for writing the sample definition down.
Read the specification out of the document before opening any data. Write the five items on one page: sample, features, fitting procedure, split or seed, metric and population. Any item you cannot fill in is a finding you already have, and you have it before you have spent any compute.
Rebuild the sample and check the row count first. 4,000 development applications at a bad rate of 0.292. If either number disagrees with the document, stop here. Every later comparison is meaningless until the two samples are the same object.
Fit from the written feature list, in your own code. Not their script. The point of the exercise is to test whether the written list is sufficient, and their script silently supplies everything the list left out.
Compare to the same precision the document quotes. The document says 0.863, so compare at three decimal places. Comparing at more precision than was reported invents a disagreement; comparing at less hides one.
Record the result either way, with the date and the specification you ran. A pass is one line in the evidence section. A gap is a finding with the list from Part 2 attached and your position on it.
Change exactly one undocumented thing and re-run. Here it is the sample window. One change, everything else held, and the size of the move tells you how much the missing item was worth. This is also the evidence that turns "please document the window" into a condition somebody acts on.
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
# Everything below is taken from the model document and nothing else.
dev = pd.read_csv("brackwater_dev.csv") # sample: 4,000 development applications
y = dev["bad"] # outcome: dev["bad"].mean() is 0.292
champion = ["util", "priorDelq", "employed", "incomeRat", "thin",
"age", "collections", "n0", "n1", "n2", "n3"] # the documented feature list
m = LogisticRegression(max_iter=2000, C=1e4).fit(dev[champion], y)
auc = roc_auc_score(y, m.predict_proba(dev[champion])[:, 1])
print(round(auc, 3)) # 0.863, the figure in the model document
print(round(auc, 3) == 0.863) # True: it reproduces exactly
# One line for the evidence section:
# "Reported development AUC of 0.863 independently reproduced from the
# documented sample, feature list and fitting procedure."# Same specification. One thing changed, and the document does not record it:
# which window the applications were drawn from.
oot = pd.read_csv("brackwater_oot.csv") # 2,000 applications, the following year
no_leak = [c for c in champion if c != "collections"]
m2 = LogisticRegression(max_iter=2000, C=1e4).fit(dev[no_leak], y)
on_dev = roc_auc_score(y, m2.predict_proba(dev[no_leak])[:, 1])
on_oot = roc_auc_score(oot["bad"], m2.predict_proba(oot[no_leak])[:, 1])
print(round(on_dev, 3)) # 0.772
print(round(on_oot, 3)) # 0.682
print(round(on_dev - on_oot, 3)) # 0.090, and not one line of the model changed
# A document that does not state its window can be quoting either number,
# and a reader has no way to tell which. That is the finding, not the 0.090.Keep the two fits side by side when you write it up. The first shows the document is reproducible on the sample it names. The second shows what a single unnamed choice is worth on this model. Together they turn a documentation request into something with a number attached, and a documentation request with a number attached is the only kind that gets actioned.
Three questions
Your re-run of Brackwater's champion reproduces the documented development AUC of 0.863 exactly. What belongs in the report?
A passing replication establishes that the document describes the model that actually exists, which every later finding in the report depends on and which nobody had checked. It says nothing about whether 0.863 is a good number, whether it survives a later vintage, or whether it was earned by a field written after the outcome, so C claims far more than the test supports.
Your re-run comes out below the documented figure. Before contacting the developer, what is the first thing to check?
The four causes are worked in order of likelihood and cost: a different feature set or sample first, then an undocumented preprocessing step, then an unrecorded seed or split, then a metric computed over a different population. An unrecorded seed has a distinctive signature, a gap that changes size on every re-run, so it is cheap to rule in later. Arithmetic is almost never the cause, and treating it as one is how a validator loses the cooperation needed to close the gap.
Under the revised interagency guidance of 17 April 2026, what is a replication actually testing?
Sound practice subjects modelling aspects to critical analysis by evaluating both the quality and the extent of developmental evidence. Reading that evidence tests neither; re-running it tests both, because a specification too thin to reproduce has an extent that cannot be assessed and a result that does not regenerate has a quality that cannot be assessed. Note also that SR 11-7 is superseded and the April 2026 text is the current one.
Before session 5
40 minTake the documentation for a model your team runs and fill in the five items from Part 1 using only what is written there: the sample with its dates and exclusions, the feature list with transformations, the fitting procedure and its settings, the split or the seed, and the metric with the population it covers. Do not ask anybody. Whatever is left blank after thirty minutes is the replication finding for that model, and you have it without running anything. Then pick the single blank you think would move the number most, and bring your reasoning to session 5, which puts the model against a three-feature challenger and asks what the complexity was buying.