Where this sits in the practitioner track
Ten sessions, 45 minutes each, and a laptop from here on. Session 1 built the inventory and made the tiering call. Session 2 read Brackwater's design before touching a number: the purpose statement, the assumptions, the data selection and the developmental evidence offered for each. Tonight is the finding that falls straight out of that reading, and it is the single most common material finding in consumer scoring: a feature whose value is written after the thing the model claims to predict. Session 4 is replication, where the reported number gets reproduced from the data rather than believed. The worked example is Brackwater's consumer scorecard throughout, with 4,000 development applications at a bad rate of 0.292.
★ What you walk out with tonight. A definition of target leakage that does not depend on any statistic, so it survives being applied to a model you have never seen. The four questions that find one, in an order you can run down a data dictionary with. The reason the usual defence does not work: a leak is present in training and in test, so the split that catches overfitting cannot catch this. And the vocabulary to file it correctly, because a leak is a conceptual soundness finding rather than a performance finding, and writing it up as the latter is how it gets argued away.
LiveA leak is a fact about timing, not about statistics4 min▶
A leaked feature is one whose value is determined after, or as a result of, the outcome the model predicts. That is the whole definition. It says nothing about correlation, nothing about variance, nothing about a metric. You can decide whether a feature is leaked by reading the data dictionary and asking one question of each field: when is this value written, relative to the moment the prediction has to be made.
This matters because every instinct a modeller has points somewhere else. The trained reflex when a number looks too good is to suspect overfitting, and the trained response is to hold out data. That reflex is useless here. An overfitted model has learned noise in the training rows that does not repeat in the test rows, which is why a train/test gap exposes it. A leaked feature is not noise. It is a real, stable, reproducible relationship, and it is present in the training rows and the test rows in exactly the same strength, because the leak was baked into the table before anybody split it. Both sides of the split look fine. The number is honest about the data and the data is dishonest about the world.
Brackwater's model document reports a development AUC of 0.863 and a stable holdout. The holdout was stable. Every row in it carried the same post-outcome field as every row in training, so there was nothing for the split to reveal. The finding was available to anyone who read the eleventh line of the feature table and asked when that column gets populated.
Self-studyFour words that get confused with each other4 min▶
- Target leakage. A feature whose value is determined after, or because of, the outcome. A timing property of the data, found by reading the process that writes the field.
- Overfitting. A model that has fitted noise specific to the training rows. A property of the fit, found by holding data out. Different problem, different detection, different remedy.
- Train/test contamination. The same records, or records sharing a key, appearing on both sides of the split. Also a data problem, but it does leave a gap when the split is repaired, which leakage does not.
- Availability at scoring time. Whether the field will physically exist in the production feature payload at the moment the score is computed. A leaked field usually fails this too, and that failure is often what finally surfaces the leak, in production, after launch.
The practical consequence of keeping these apart: three of the four are found by running something, and the first one is found by reading something. If your leak check is a script, it is checking for the wrong thing.
Put every feature on a timeline and one of them is on the wrong side
9 min liveThere is a moment in the process when the score has to exist. For an application scorecard it is the moment the decision is made, before anything about the customer's subsequent behaviour is known. Every feature the model uses must have a value written before that moment. The outcome window opens after it, and everything the business does in response to what happens inside that window gets written later still. A leak is simply a field from the second region that has found its way into the first.
LiveHow a post-outcome field gets onto a feature list in the first place4 min▶
Nobody adds a leaked field on purpose. It arrives through a join. The modeller needs application attributes, pulls them from the application table, and then needs behavioural context, so joins the account table as it stands today. Today is well after the outcome window closed, and the account table has been accumulating rows the whole time: arrears flags, contact history, hardship markers, restructures, collections activity. The join is correct SQL against a table that is correct on its own terms. What is wrong is the as-of date, and no join key carries one.
The second route is a feature store or a curated mart built for a different purpose. Operational reporting wants the current state of an account, and it is right to want that. A training set wants the state of an account as of the decision date, which is a different object with the same column names. When the two share a catalogue and a naming convention, nothing in the feature list tells you which one you are holding.
Both routes leave the same fingerprint: the field is real, the values are correct, the lineage is documented, and the timing is wrong. A data quality check will pass it. A schema test will pass it. A reviewer who knows what the business process does with the field will catch it in under a minute.
LiveThe consequence-is-not-a-cause point, and where to stop3 min▶
The plain statement of the Brackwater finding is that a customer who defaults gets contacted, and the contact did not cause the default. The arrow runs from outcome to feature. That is enough for a validation finding, and a validator should say exactly that much and no more.
It is worth naming where the temptation goes, so you can decline it. There is a whole formal apparatus for reasoning about what causes what and about when conditioning on a variable creates a relationship that is not there, and learn-causal-inference-with-phoebe owns it. A validation report does not need it. The finding here does not rest on identifying a causal effect; it rests on the order of two timestamps, and a timestamp is not contestable in the way a causal claim is. Reaching for the causal vocabulary makes a finding that is currently unarguable into one the model owner can argue with.
Write the finding as a sentence about process, not about statistics. "The field collections counts contacts logged after the outcome window opens, so its value is determined by the outcome the model predicts." No metric appears in that sentence, which is why no metric can be produced to rebut it.
Self-studyThe coefficient is a clue, and not the evidence3 min▶
In Brackwater's fitted champion, collections carries a coefficient of 2.214, second only to utilisation's 2.612. That ordering is a useful thing to notice and a poor thing to argue from. Coefficients in a logistic fit are on the scale of the feature, so an input recorded in different units, or with a different spread, is not comparable to its neighbour without care, and a large coefficient on a genuinely predictive field is exactly what you would hope to see.
What makes the coefficient interesting is the combination with the timing fact. A field written after the outcome should not be doing anything at all, so any material weight on it is weight the model is drawing from the future. Use the coefficient to decide which fields to read the dictionary entry for first, in a list too long to read end to end. Do not put it in the finding.
The same caution applies to the per-feature scores in the next part. They are a triage device that tells you where to look. The finding is always the timing.
Score every column on its own
10 min liveReading the dictionary finds the leak. Scoring each column alone tells you how much of the model was resting on it, which is what turns a note into a material finding. Three numbers make the Brackwater case, and they can be produced in about twenty lines. The champion as submitted scores 0.863 on development. The collections column alone scores 0.748. The four pure-noise features together score 0.520. One post-outcome field carries almost all of the apparent skill, and ten others carry nearly nothing.
import pandas as pd
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score
dev = pd.read_csv("brackwater.csv") # 4,000 development applications
y = dev["bad"] # dev["bad"].mean() is 0.292
champion = ["util", "priorDelq", "employed", "incomeRat", "thin",
"age", "collections", "n0", "n1", "n2", "n3"]
def auc_of(cols):
"""Fit on these columns only, score the same sample, return the AUC."""
X = dev[cols]
m = LogisticRegression(max_iter=2000, C=1e4).fit(X, y)
return roc_auc_score(y, m.predict_proba(X)[:, 1])
print(round(auc_of(champion), 3)) # 0.863, the number in the model document
for col in champion:
print(col.ljust(12), round(auc_of([col]), 3))
# collections 0.748 <- one column, against a whole model at 0.863LiveWhy it survived review: it is noisy, not a copy of the target4 min▶
The reason this leak lived through a build, a review and a sign-off is the most instructive thing in the whole case. Of Brackwater's defaulters, 58 percent get a collections contact. Of the non-defaulters, 11 percent get one. The field is strongly related to the outcome and nowhere near a copy of it.
Consider what the alternative would have looked like. A field that reproduced the target exactly would have produced an AUC at or near the ceiling, and a modeller seeing that reaches for the join immediately, because an almost-perfect model is a universally recognised alarm. Nobody ships that. What Brackwater had instead was a field that lifted the model into a range that looked like a good build by a competent team. There is no alarm for that. Good numbers are the thing everyone is trying to produce.
The generalisation is uncomfortable and worth saying out loud: the leaks that get caught are the useless ones. The leaks that reach production are the ones that are only partially informative, because partial information is indistinguishable from a well-engineered feature when all you look at is performance. Which is the argument for not looking at performance first.
The Brackwater review did notice the collections field. It was raised, discussed, and recorded as an observation rather than a condition, on the grounds that the field was "predictive and available in the warehouse". Both of those statements are true. Neither is relevant to the question of when the value is written.
LiveFile it as conceptual soundness, not as performance3 min▶
Validation has three components, and the current guidance puts conceptual soundness first. It covers model design, key modelling choices, assumptions, qualitative judgements and data selection, and the way it is assessed is by critically evaluating the quality and extent of the developmental evidence. Read that list again with a leaked feature in mind: data selection is in it, explicitly. A post-outcome field in a feature list is a data selection defect, assessed by reading the evidence the developer offered for including it.
Filing it anywhere else weakens it. As a performance finding it becomes a debate about how much the number moves, which is a debate the model owner can win with a business argument about the cost of the removal. As a data quality finding it goes to whoever owns the warehouse, and the warehouse is not wrong. Conceptual soundness is the category that says the model as designed does not do what it claims, at which point the number is beside the point.
This is also the category that survives a rebuttal of the form "the field is available in production". Sometimes it genuinely is, because the scoring pipeline reads the same current-state table. That makes the operational risk worse rather than better: the model would then be scoring new applicants on a field that is structurally empty for them and populated for anyone with history, which is a different defect on top of the first one.
Self-studyWhat the ten other features were doing3 min▶
The per-column sweep answers a second question for free. Four of Brackwater's eleven features are pure noise by construction, and fitted together they score 0.520. That is a useful control to have in the same table as the headline, because it calibrates the reader's eye: it shows what a column contributing nothing looks like in this exact setup, on this exact sample, with this exact fitting procedure.
A validator should want that control in every sweep, and it does not require the data to have been generated. Permuting a column and refitting gives the same service. The point is to have a floor on the page, so that a reviewer looking at a per-feature table can tell the difference between a weak feature and a decorative one without appealing to a rule of thumb.
What the sweep does not license is a recommendation to drop the ten. Simplification is a modelling decision, and the report's job is to state what the evidence shows rather than to redesign the model. The benchmark that tests whether the complexity earns its place is session 5's job, and it is a different exercise with a different standard of proof.
Run the hunt on the model in front of you
18 min build-alongThe checklist below is the whole technique, and it generalises past credit scoring to any model with an outcome window. Run it against Brackwater tonight, then run it next week against a feature list from your own book. Steps 1 and 2 are reading. Steps 3 and 4 are code. Step 5 is the part people skip and then discover at launch.
Fix the prediction moment, in writing, before you look at anything. One sentence: "the score must exist at the point the application is decided, before any repayment behaviour is observed." Everything else in the checklist is measured against this sentence, so if you cannot write it the model has a purpose problem that outranks the leak hunt.
For every feature, ask when its value is known relative to that moment. Every feature, including the boring ones. The answer has to come from the data dictionary or from a person who knows the process, not from the feature name. Fields whose answer is "when the record was last updated" are unresolved, not clean.
Ask what process writes each field, and what triggers that process. This is the question that catches the ones timing alone misses. A field can be written before the decision and still be a consequence, if the thing that triggers the write is correlated with the outcome. "Which team populates this, and what makes them populate it" is the form of the question that gets a useful answer.
Score each feature alone and be suspicious of any single field that approaches the full model. Not as proof, as triage. Anything within striking distance of the headline goes to the top of the reading list for steps 2 and 3. Include a floor, either genuinely uninformative columns or permuted ones, so the table has a reference point.
Check whether the feature would even exist at scoring time in production. Ask for the production feature payload, not the training table. A field that is present in one and absent or structurally empty in the other is either a leak or a live incident waiting for the first score, and often both.
Refit without the suspect field and quote both numbers. The submitted number and the number without the field, side by side, with the difference named. This is what converts a conceptual point into a material one, and it is what the approver needs in order to understand what was actually being bought.
Write it as a finding about data selection. State the timing fact, state the process that writes the field, state the effect on the reported number, and stop. Resist adding a recommendation about what should replace it; that is the developer's job and putting it in your finding gives them something to negotiate over.
no_leak = [c for c in champion if c != "collections"]
noise = ["n0", "n1", "n2", "n3"]
full = auc_of(champion) # 0.863 as submitted
clean = auc_of(no_leak) # 0.772 with the field removed
print(round(full, 3), round(clean, 3))
print(round(full - clean, 3)) # 0.091, what the leak was worth
print(round(auc_of(noise), 3)) # 0.520, the floor: four noise columns
print(round(auc_of(["collections"]), 3)) # 0.748, the column on its own
# Both numbers go in the report. 0.863 is what was claimed;
# 0.772 is what the model does using only fields that exist at decision time.# A field that predicted perfectly would have been obvious.
# This one only predicts well, which is what a good feature looks like.
contacted = dev["collections"] > 0
print(round(contacted[y == 1].mean(), 2)) # 0.58 of defaulters were contacted
print(round(contacted[y == 0].mean(), 2)) # 0.11 of non-defaulters were
# Weight in the fit: a clue for triage, never the finding itself.
m = LogisticRegression(max_iter=2000, C=1e4).fit(dev[champion], y)
coef = pd.Series(m.coef_[0], index=champion).sort_values(ascending=False)
print(coef.round(3).head(2))
# util 2.612
# collections 2.214
# The finding is still the timestamp, not either of these numbers.The question that settles it in one exchange. Ask the developer to describe, step by step, what has to happen in the business for this field to take a non-zero value. If any step in their answer involves the customer having already done the thing the model predicts, the hunt is over and you have the wording of your finding in their own words.
Three questions
A model owner responds to your leakage finding by showing that training and holdout AUC are nearly identical, and says this rules out a leak. What is wrong with the argument?
The split catches a model that has fitted noise specific to the training rows. A post-outcome field is a stable, reproducible relationship that was baked into the table before it was split, so both sides carry it identically. Matched numbers are exactly what a leak produces, which is why the reassurance is worthless here.
Brackwater's champion scores 0.863 on development. The collections field alone scores 0.748, and the four pure-noise features together score 0.520. What is the defensible reading?
The sweep is a statement about the model document, not a verdict on the model. Refitting without the field gives 0.772, so there is a real model underneath. The finding is that 0.863 was quoted for something that partly measures the future, and the floor at 0.520 is there to show what a column contributing nothing looks like, not to trigger a simplification recommendation.
Under the revised interagency guidance of 17 April 2026, which component of validation does a target leakage finding belong to, and why does the category matter?
Data selection sits inside conceptual soundness, and that component is assessed by critically evaluating the quality and extent of the developmental evidence, which is precisely what a leaked field fails. The AUC sweep is triage that sizes the finding; the finding itself is that the field is written after the outcome, and that is a design and data selection defect. Note that SR 11-7 is superseded and the April 2026 text is the current one.
Before session 4
40 minTake a feature list from a model your team actually runs, and answer step 2 for every single field: when is this value written, relative to the moment the score has to exist. Write the answer next to each field, and write "unknown" where you cannot find out, because the unknowns are the finding. Then run the per-feature sweep on the same model and put the two columns side by side. Bring the table to session 4, which is about replication: reproducing the number the model document reports, and what it means when the re-run and the document disagree.
What tonight maps to
collections alone 0.748; the four pure-noise features together 0.520; refit without the field 0.772, so the leak was worth 0.091. Fitted coefficient 2.214, second only to utilisation at 2.612.