learn-ml-epistemology-with-phoebe / Practitioner session 2 of 10
Learn ML Epistemology with Phoebe · Practitioner session 2 of 10

Building the shortcut and measuring reliance

Session 1 ended on a sentence with nothing to back it up yet: a held-out score cannot see a shortcut, because the shortcut works on the held-out rows too. Tonight that stops being an argument and becomes a measurement. Flip the ward on every patient, hold everything else fixed, and count how many predictions change. At Thornbury that share runs from 0.70 to 1.00 depending on nothing more than the random seed, and every one of those models scores about 0.949 on the held-out set.

🔴 Hardest Anyone who has been asked which feature the model is using Anyone who has answered by reading a coefficient 45 min
0-3 · Welcome 3-17 · Shortcut, not leak 17-31 · Behavioural reliance 31-45 · Build-along and Q&A
Part 0

Where tonight sits

Session 1 built the Thornbury world and split it two ways, and established what a held-out number is an estimate of. Tonight trains a model in it and asks a question the held-out number cannot answer: how hard does this particular model lean on the feature that will stop working? That question has an answer you can compute in four lines, and the answer varies wildly between models the score says are identical. Session 3 turns that into twenty models and a spread. Session 4 designs the world to test them in.

Live = written together in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. A clean line between a shortcut and a leak, which matters because the two get different fixes: one is measured and stress tested, the other is removed and the validation re-run. A four-line reliance measure you can point at any binary feature in any model, with the Thornbury range 0.70 to 1.00 as the reference for how far apart two equally scored models can sit. And the reason weight inspection does not answer this question once there are hidden units in the way.

LiveA shortcut is not a leak, and the difference decides the fix5 min▶

The ward a patient is on is recorded at admission. It is sitting in the record hours before anybody knows whether that patient deteriorates, and it would be sitting there at prediction time in production too. Nothing about it is unavailable, retrospective or contaminated. It is not leakage.

What it is instead is a feature whose usefulness depends on an arrangement. In the collection year the high-acuity ward received most of the patients who went on to deteriorate, so the ward agreed with the outcome about 90 percent of the time without causing it. The agreement is a fact about how the hospital group was organised that year, and hospital groups reorganise. The mechanics of the other failure, how a post-outcome field reaches a feature list and why a noisy one survives review, belong to learn-model-risk-with-phoebe p3.

Getting this line right is not pedantry, because the two failures get opposite treatments. A leak is removed and the validation is re-run, and the number that comes back is the real one. A shortcut is not removed by default: it was genuinely predictive, a model that drops it scores worse, and the honest move is to measure how much the model depends on it and then test the model where the arrangement no longer holds.

Self-studyThe words that will keep coming back4 min▶
  • Shortcut. A feature that predicts the outcome without causing it, whose usefulness depends on an arrangement that can change.
  • Leakage. Information in the feature list that could not have been known when the prediction was made. Different failure, different fix, and the mechanics live in learn-model-risk-with-phoebe p3.
  • Behavioural reliance. The share of rows whose predicted class changes when one feature is flipped and everything else is held fixed. Tonight's measurement.
  • Intervention. Setting a feature to a value rather than observing it. The reliance measure is an intervention on the input, which is why it answers a question the held-out score cannot.
  • Decay against reversal. A shortcut can stop being informative, which is Thornbury's case, or it can start pointing the other way. The second is worse and is not modelled here.
  • The standing pair. Mean held-out 0.949 and mean shifted 0.765 across the five data draws. Every session quotes those two numbers against each other.
Part 1 · covers shortcut against leak

Available at prediction time, and still a problem

12 min live

The clock settles the question. Everything known before the outcome is fair game as a feature, and being fair game is not the same as being durable. The two categories look identical in a feature importance plot and need completely different responses, so the clock check comes first.

A shortcut is available when the prediction is made. A leak is not. Admission Prediction made Outcome known, 24 hours later Ward: recorded at admission known before the outcome, so not a leak A post-outcome field lands here could not have been known in time Shortcut Real, usable, honestly available at prediction time. Its usefulness depends on an arrangement in the world. Fix: measure the reliance tonight, then stress test it in session 4. Leak The field could not have been known when the prediction was made. Fix: remove it and re-run the validation. Mechanics in learn-model-risk-with-phoebe p3.
🔍 Click to zoom - same suspicious feature importance, two different failures, two different fixes.
LiveWhy removing the ward is not the obvious answer4 min▶

The instinct in the room is always to drop the feature. Resist it for a minute. In the collection year the ward carried genuine information about who was going to deteriorate, because the allocation policy put the sicker patients in the high-acuity ward. A model that never sees the ward scores lower on the held-out set, and in a normal review it loses to the model that does see it. Dropping a feature because it makes you uncomfortable also costs you the ability to notice when the arrangement changes, since the model no longer responds to it at all.

The defensible sequence is narrower. Establish that the feature is available in time, which makes it a feature rather than a leak. Measure how much this model depends on it, which is tonight. Then score the model where the arrangement has changed, which is the shifted set and session 4. Removal is one possible outcome of that sequence, not its starting point.

LiveThe standing pair: 0.949 and 0.7654 min▶

Across the five data draws the mean held-out accuracy is 0.949 and the mean shifted accuracy is 0.765. Those two numbers are quoted against each other for the rest of the track, so it is worth being exact about what separates them. The model is the same. The architecture is the same, three inputs into twelve tanh hidden units into one logit. The metric is the same. The only difference is which world the 2,000 scored patients came from: the collection year, where the ward agrees with the outcome 90 percent of the time, or the deployment year, where it agrees 50 percent of the time.

Real world

When a stakeholder asks what the model is worth, both numbers are the honest answer and neither one alone is. The first says what it does where it was measured. The second says what one changed arrangement costs. A review that hears only the first has been given a true statement about a world that has an expiry date on it.

Part 2 · covers behavioural reliance

Flip the ward and count what moves

12 min live

This is the measurement the session owns, and it is almost embarrassingly simple. Take every patient in the held-out set. Score them once with the ward set to 0 and once with it set to 1, changing nothing else. Count the fraction whose predicted class differs between the two. That fraction is the model's behavioural reliance on the ward, and it is a fact about the model rather than about the data it was scored on.

Flip the ward, hold everything else fixed, count the predictions that change. The same patient, scored twice vitals and lab as recorded, ward = 0 predicts stable the very same row, ward = 1 predicts deterioration One model's reliance on the ward changed predictions divided by every patient scored 0.70 to 1.00 across twenty seeds at Thornbury Twenty models, one pipeline, one difference between them: the random seed. measured range, 0.70 to 1.00 0.00 0.50 1.00 share of patients whose prediction changes when only the ward moves Every one of these models scores about 0.949 on the held-out set.
🔍 Click to zoom - one intervention, one count, and a range the held-out score cannot see.
LiveWhat a range of 0.70 to 1.00 actually says4 min▶

At the bottom of the range, a model changes its mind about 7 patients in 10 when the only thing that moves is the ward. At the top, it changes its mind about all of them: the ward alone decides the prediction for every patient in the set. These are not different pipelines, different features or different data. They differ by the random seed, which sets the initial weights and the order the examples arrive in, and nothing else.

Now put the held-out score next to that. Every one of those models scores about 0.949. The number everybody in the review will look at is constant across a range of behaviour that runs from heavy dependence to total dependence. That is why the reliance measure is worth four lines of code: it is the only thing in the room that distinguishes the models, and it distinguishes them before the arrangement changes rather than after.

LiveWhy reading the weights does not answer this4 min▶

With a single linear layer you could read the coefficient on the ward and argue about its size. Thornbury's model has twelve tanh hidden units between the input and the output, so the ward's influence is spread across twelve incoming weights and then recombined through twelve outgoing ones. There is no single number to read, and the product of the two layers is not the answer either, because the tanh is not linear and the hidden units interact.

The flip sidesteps all of it. Rather than asking what the weights mean, it asks what the model does, which is the question anybody actually had. It also stays honest when you replace the network with a gradient-boosted tree, an ensemble or something nobody can read at all: the measurement only needs predictions with the feature set two ways.

Self-studyHow the deployment world is designed, and the case it does not include4 min▶

The deployment year at Thornbury is built by one change: the ward agreement drops from 90 percent to 50 percent. The shortcut does not disappear from the feature list, it does not become missing, and it does not turn hostile. It decays into noise, which is the mild version of this failure, and the honest label for a feature agreeing with the outcome half the time is uninformative.

The worse case exists and is deliberately not modelled: a shortcut that reverses, where the arrangement flips so the feature now points away from the outcome. A model leaning hard on a reversed shortcut is confidently wrong rather than merely unhelpful, and the drop is larger than anything shown here. Take 0.765 as the cost of decay, not as a worst case, and say so when quoting it.

Part 3 · build-along

Train one model, then interrogate it

13 min build-along

Two blocks, both on top of the generator from session 1. The first trains one small network on the collection year and scores it twice, where it was measured and where the arrangement has changed. The second is the reliance measure, which is four lines and a loop.

Import last session's generator. The one with two standard-normal signals, the logit and the ward that agrees with the outcome with probability p. Nothing tonight works without it.

Train one network on the collection year. Three inputs, twelve tanh hidden units, one logit, trained by stochastic gradient descent. Small on purpose: nothing tonight is about capacity.

Score it twice. Once on the held-out rows from the collection year, once on the shifted rows from the deployment year. Expect the neighbourhood of 0.949 and 0.765 rather than those digits exactly.

Write the reliance function. Copy the feature matrix twice, set the ward column to 0 in one copy and 1 in the other, predict both, and take the mean of the disagreement.

Run it across seeds. Change only the random seed and watch the reliance move while the held-out score does not. That contrast is the whole session in one printout.

Point it at something of your own. Take the binary feature you named in last session's homework and run the same four lines. The number you get is the one your next review should hear.

★ Block 1 · one model, scored where you looked and where you did not import numpy as np from sklearn.neural_network import MLPClassifier # make_data is the Thornbury generator written in session 1 X_train, y_train = make_data(4000, 101, 0.90) X_held, y_held = make_data(2000, 102, 0.90) X_shift, y_shift = make_data(2000, 103, 0.50) def thornbury_net(seed): return MLPClassifier(hidden_layer_sizes=(12,), activation="tanh", solver="sgd", learning_rate_init=0.08, alpha=2e-4, max_iter=30, random_state=seed) net = thornbury_net(1).fit(X_train, y_train) print("held out:", round(net.score(X_held, y_held), 3)) # around 0.949 print("shifted :", round(net.score(X_shift, y_shift), 3)) # around 0.765 # This is a different random number generator from the engine that produced # the canon numbers on this site, so the digits here will not reproduce them # exactly. The neighbourhood is the claim; the last decimal is not.
★ Block 2 · behavioural reliance on the ward import numpy as np WARD = 2 # the third column of X: vitals, lab, ward def reliance_on_ward(model, X): """Flip the ward on every patient, hold vitals and lab fixed, and return the share of predictions that change.""" X0 = X.copy(); X0[:, WARD] = 0 X1 = X.copy(); X1[:, WARD] = 1 return float(np.mean(model.predict(X0) != model.predict(X1))) print("reliance:", round(reliance_on_ward(net, X_held), 2)) # the same pipeline, the same data, only the seed moving for seed in range(1, 6): m = thornbury_net(seed).fit(X_train, y_train) print(seed, "held out", round(m.score(X_held, y_held), 3), "reliance", round(reliance_on_ward(m, X_held), 2)) # Across twenty seeds at Thornbury the reliance runs from 0.70 to 1.00, # and every one of those models scores about 0.949 on the held-out set. # Session 3 runs the full twenty and measures the spread that follows.

The habit to keep. For any binary feature somebody is nervous about, reliance is two copies of the matrix and one mean. Report it next to the accuracy, not instead of it, and report it before anybody asks. A model at 0.949 with a reliance of 1.00 on one arrangement-dependent feature is a different object from a model at 0.949 with a reliance of 0.15, and the score alone will never say which one is on the slide.

Check yourself

Three questions

The ward is recorded at admission, hours before anybody knows whether the patient deteriorates. What is it?

Leakage is information that could not have been known in time, and the ward fails that test: it was there at admission and would be there in production. The problem is durability rather than availability, which is why the fix is to measure reliance and stress test rather than to remove and re-run.

Twenty Thornbury models differ only by their random seed, and their reliance on the ward runs from 0.70 to 1.00. What do their held-out scores look like?

The shortcut works on the held-out rows too, so leaning on it neither helps nor hurts the number that gets reported. The models are indistinguishable where everybody looks and range from heavy to total dependence underneath. Session 3 measures the spread this produces once the arrangement changes.

Why measure reliance by flipping the ward rather than by reading the model's weight on it?

The ward's influence is spread over twelve incoming weights, recombined through twelve outgoing ones, and passed through a nonlinearity, so there is no number to read. The flip asks what the model does instead of what its parameters mean, and it keeps working on a model nobody can inspect at all.

Homework

Before session 3

30 min

Run tonight's two blocks over ten seeds instead of five and keep the printout: held-out accuracy and reliance side by side, one row per seed. Look at the two columns and write one sentence about what varies and what does not. Then write down, in advance, what you expect to happen to those same ten models on the shifted set. Session 3 runs twenty seeds and measures both spreads, and a prediction you wrote before the run is worth more than any amount of agreement afterwards.

Sources covered

What tonight maps to

✓ Shortcut against leak The ward is available before the outcome is known, so it is a feature whose usefulness depends on an arrangement, not leakage. The two failures get different fixes.
✓ Behavioural reliance, measured Flip the ward on every patient, hold everything else fixed, count the changed predictions. Across twenty seeds at Thornbury this runs from 0.70 to 1.00 while held-out accuracy stays at about 0.949.
✓ Why weight inspection does not answer it With twelve tanh hidden units there is no single coefficient for the ward; the behavioural measure answers the question directly and survives an unreadable model.
✓ How the deployment world is designed The shortcut decays from 90 percent agreement to 50 percent rather than vanishing. A shortcut that reverses would be worse and is not modelled.
✓ The standing pair Mean held-out 0.949 against mean shifted 0.765 across the five data draws, quoted together for the rest of the track.
◐ Twenty seeds and the spread that follows Session 3 runs the full bench: identical held-out scores, divergent shifted scores, and the ratio between the two spreads.
◐ Designing the shifted set Session 4. What a stress test can and cannot establish.
· Leak mechanics inside one validation Not covered by design. learn-model-risk-with-phoebe p3 owns timing, how a post-outcome field reaches a feature list, and filing it as a validation finding.
· Memorization, random labels and double descent Not covered by design. learn-deep-learning-with-phoebe session 05 owns how a network can fit noise and what the interpolation threshold means.
· Metric definitions and drift detection Not covered by design. learn-model-evaluation-with-phoebe defines every metric and owns monitoring.

Session 2 cheat sheet

The clock decidesKnown before the outcome means feature, not leak. Availability and durability are different properties.
Different fixesA leak is removed and the validation re-run. A shortcut is measured, then stress tested.
Reliance in four linesCopy the matrix twice, set the feature to 0 and to 1, predict both, take the mean disagreement.
0.70 to 1.00The reliance range across twenty seeds. Same data, same architecture, only the seed differs.
All of them score 0.949The held-out number is constant across that whole range, which is why it cannot rank them.
Weights will not tell youTwelve hidden units and a nonlinearity leave no coefficient to read. Ask what the model does.
Decay, not reversalAgreement falls from 90 percent to 50 percent. A shortcut that reversed would cost more and is not modelled.
Report it unpromptedAccuracy plus reliance. Two models at 0.949 with reliance 1.00 and 0.15 are not the same object.