learn-ml-epistemology-with-phoebe / Leader session 1 of 6
Learn ML Epistemology with Phoebe · Leader session 1 of 6

What a test score entitles you to

Thornbury's deterioration-risk model scores 0.949 on data it never trained on. That number is correct, it reproduces, and it was computed properly. A year later the hospital group reorganises its wards and the same model, unchanged, scores 0.765. Nothing was retrained and nobody made a mistake. The number was never a fact about the model; it was a claim about a future that would resemble the past, and one of the resemblances quietly expired. Tonight is about what that kind of number does and does not entitle you to say.

🔴 Hardest Whoever approves the deployment Anyone who has been handed an accuracy figure 45 min
0-3 · Welcome 3-20 · What the number claims 20-40 · Build-along: audit a warrant 40-45 · Q&A
Part 0

How the leader track works

Six sessions, 45 minutes each, no code. Tonight: what a held-out score is evidence of, and the three assumptions it rests on. Session 2 is underspecification, which is what happens when many models are equally good on paper and different in the world. Session 3 is what goes wrong when a number becomes a target. Session 4 is what a benchmark licenses you to claim about your own problem. Session 5 is a whole literature that was wrong in the same way, and what to do about evidence you did not produce. Session 6 is the question to ask before you approve anything: what would change your mind. The example throughout is Thornbury, a hospital group predicting which patients deteriorate within 24 hours.

Live = presented in the 45 minutes Self-study = the full depth, after class 🔍 Every diagram zooms

★ What you walk out with tonight. The three assumptions that turn a held-out number into a claim about tomorrow, so you can ask which one is load-bearing for the model in front of you. The measured cost of one of them expiring at Thornbury: 0.949 down to 0.765, with no retraining and no error. And a habit that costs one sentence in a review: name the resemblance the score depends on, before the score is quoted.

LiveA score is a claim, not a property4 min▶

When a model reports 0.949 on held-out data, the thing being reported is not a property of the model in the way that its file size is. It is the outcome of a bet: that the data it will meet resembles the data it was scored on, in the ways that matter. Held-out accuracy is the best evidence we have for that bet, and it is still evidence about a resemblance rather than a measurement of the future.

This sounds like philosophy until the resemblance breaks, at which point it becomes an incident. The useful move is to make the bet explicit while everyone is calm: say out loud which resemblance the number depends on, and who would notice if it stopped holding. Most review meetings never do this, which is why the first sign of trouble is usually a dashboard rather than a question.

Real world

Thornbury's model was scored on patients from the collection year and approved on a held-out accuracy of 0.949. In that year the high-acuity ward received most of the patients who went on to deteriorate, so the ward a patient was on agreed with the outcome about 90 percent of the time without causing it. The model found that agreement and used it. When the group reorganised its wards, the agreement fell to about 50 percent and the same model scored 0.765.

Self-studyThe words that will keep coming back4 min▶
  • Held-out set. Data kept aside from training and used once to estimate how the model behaves on data it has not seen.
  • Warrant. The reason a claim is believable. A score is not a warrant on its own; the warrant is the score plus the assumptions that let it travel.
  • Shortcut. A feature that predicts the outcome without causing it, and whose usefulness depends on an arrangement that can change.
  • Distribution shift. The world the model meets stops resembling the world it was scored on.
  • Stress test. Scoring a model on data chosen to break a specific assumption, rather than on more of the same data.
  • Underspecification. A pipeline that can return many models with equally strong held-out performance which behave differently in deployment. Session 2.
  • Falsifier. The observation that would make you stop believing the claim. Written down in advance, it is the cheapest control there is. Session 6.
Part 1 · covers the three assumptions

Three assumptions, and one of them is usually the problem

10 min live

A held-out score travels from the test set to next Tuesday along three links. Each one is an assumption, each one can break independently, and they break for different reasons. Naming which link is load-bearing takes about a minute and changes what you monitor.

The score is measured here and spent there. Three assumptions carry it. The test set 0.949 at Thornbury 1 · The same world Tomorrow resembles the test set 2 · An untouched draw The set did not pick the model 3 · The right target Measured what the decision needs How each one actually breaks A shortcut expires The population moves The process changes Seasons, policy, a new form The set was scored twice A leaderboard was climbed Thresholds tuned on it Session 3 measures the cost Accuracy, not harm Average, not the ward A proxy for the outcome Session 4 is the benchmark case Thornbury broke the first one, and only the first one. The score was computed correctly on an untouched set that measured the right thing. It still stopped being true.
🔍 Click to zoom - three links, three different failure modes. Only one of them is anybody's mistake.
LiveWhy the first link is the one that usually goes4 min▶

Links two and three are under somebody's control. A test set can be kept clean by discipline, and the target can be chosen carefully by argument. The first link is not under anybody's control, because it is a claim about a world that keeps moving for reasons that have nothing to do with the model: a reorganisation, a new intake process, a supplier change, a policy, a season.

That asymmetry is worth saying out loud in a review, because it changes what you ask for. For links two and three you ask about process. For link one you ask a different question: what in this score depends on an arrangement that could change, and who would tell us if it did. A model that leans on something stable can survive neglect. A model that leans on a current arrangement needs a watcher.

Self-studyWhy "more data" does not repair link one3 min▶

A bigger test set makes the estimate of today's accuracy more precise. It says nothing about whether today resembles tomorrow, because every extra row is drawn from the same today. Precision and validity are different properties, and the first is much easier to buy.

This is the reason a very large held-out set can be reassuring and useless at the same time. The honest reading of a tight estimate is "we know what this model does on data like this, to within a small margin", which leaves the entire question of what data will be like completely open.

Part 2 · covers the Thornbury drop

What one expired assumption cost

8 min live

The Thornbury numbers are worth sitting with because nothing in the story is a mistake. The data were clean, the split was honest, the metric was reasonable, and the model was competently built. The drop happened anyway, and the size of it is the argument for everything in the remaining five sessions.

The same model, scored twice. No retraining between them. Collection year, held out 0.949 ward agrees 90% of the time Deployment year, wards reorganised 0.765 ward agrees 50% of the time A fall of 0.184, and not one line of the model changed. The model learned an arrangement, not a mechanism. The arrangement was real, it was predictive, and it was never promised to anybody. Nothing in the held-out score could have told you which kind it had learned.
🔍 Click to zoom - the drop is the price of one assumption, measured.
LiveThe shortcut was not a bug, and that is the uncomfortable part4 min▶

It is tempting to file this as a data quality problem and ask for the ward to be removed. That instinct is worth resisting for a moment, because the ward was genuinely informative in the collection year. A model that ignored it would have scored worse and been rejected in review for being worse. The pipeline did what it was asked to do.

The real lesson is narrower and harder: predictive and durable are different properties, and a held-out score measures only the first. Nothing in the review process at Thornbury asked which one the model had found, and no amount of additional held-out data would have answered it, because the shortcut worked on held-out data too.

Real world

Ask a modelling team what their model would do if one named feature were shuffled. At Thornbury, flipping the ward on every patient changes the prediction for between 70 and 100 percent of them, depending on which model you ask, and every one of those models scored about 0.949. That range is session 2's subject, and it is invisible from the score.

Self-studyWhat the remaining five sessions add3 min▶
  • Session 2. Twenty models from the same pipeline, equally good on held-out data, measurably different once the ward moves. That is underspecification, and it means "we validated it" does not identify which model you deployed.
  • Session 3. What happens to a number once people start selecting against it, measured rather than asserted, and why the folklore overstates it in one direction and understates it in another.
  • Session 4. Benchmarks: what a published ranking licenses you to expect on your own problem, which is less than it appears.
  • Session 5. A survey that found leakage in 294 papers across 17 fields, and what it means when the evidence you are relying on was produced by somebody else.
  • Session 6. The falsifier: writing down what would change your mind, before the result arrives.
Part 3 · build-along

Audit the warrant on a model you already trust

20 min build-along

No laptop. Pick a model your area already relies on, one whose number somebody quoted to you with confidence, and walk the three links out loud. The goal is not a verdict. It is to find which link nobody in the room can speak to, because that is the one that will surprise you.

Say the number and where it came from. Not "it is accurate" but the figure, the data it was computed on, and when. If nobody in the room can say when, you have already found something.

Name the resemblance. Finish this sentence: "this number holds as long as ___ keeps being true." Population, process, season, supplier, a policy, a form that people fill in the same way.

Ask what the model would do if one feature were shuffled. Pick the feature somebody would call the most predictive. If nobody knows the answer, nobody knows what the model learned.

Check link two honestly. How many times has that test set been looked at? Was a threshold tuned on it, a variant chosen with it, a decision made twice? Nobody is in trouble; the number is just softer than it looks.

Check link three out loud. Does the metric measure the thing the decision needs, or the thing that was easy to compute? Say who is harmed when the model is wrong in each direction.

Write the watcher. For the resemblance you named in step two, say who would notice if it stopped holding, and how long that would take. "Nobody" and "a quarter" are both common and both answers worth having in writing.

The one-sentence version to take into a meeting. "This model scores X on data from [when], and that number holds as long as [resemblance] keeps being true; [person] would notice within [time] if it stopped." A team that can say that sentence has done the epistemology. A team that cannot has an accuracy figure and a hope.

Check yourself

Three questions

Thornbury's model fell from 0.949 to 0.765 after the wards were reorganised. What went wrong?

The split was honest and the number reproduced. The model found a feature that was genuinely predictive in the collection year and was never promised to stay that way. This is not overfitting, which is a failure to generalise from training to held-out data; here the model generalised to held-out data perfectly well.

A team offers to quadruple the size of the held-out set to increase confidence before deployment. What does that buy?

Every additional row is drawn from the same world, so precision improves and validity does not. A tight estimate and a fragile assumption sit together comfortably, which is what makes a large held-out set quietly reassuring.

Which question does the most work in a five-minute model review?

The accuracy is already in the deck and the algorithm rarely determines the failure. The first question is the only one of the three that surfaces the assumption carrying the number, and it produces a named owner and a detection time rather than a reassurance.

Homework

Before session 2

30 min

Take the model you audited tonight and find out one thing: how many models were trained before that one was chosen, and what separated them. Ask the person who built it. The answer is often "a few dozen, and we kept the best on the validation set", which is the honest and normal answer, and it is exactly the situation session 2 is about. Bring the number.

Sources covered

What tonight maps to

✓ A held-out score as a claim rather than a property The three assumptions that carry it from the test set to a decision, and the different ways each one breaks.
✓ The Thornbury drop, measured 0.949 held out against 0.765 once the ward agreement falls from 90 percent to 50 percent, with no retraining in between.
✓ Predictive against durable Why a shortcut is not a bug, why a held-out score cannot distinguish the two, and why more held-out data does not help.
◐ Underspecification Session 2. Tonight names the 70 to 100 percent reliance range; the twenty models behind it come next.
◐ What selecting on a test set costs Session 3 measures link two rather than asserting it.
· Memorization, random labels and double descent Not covered by design. learn-deep-learning-with-phoebe session 05 owns how a network can fit noise and what the interpolation threshold means.
· How to compute accuracy, and drift detection Not covered by design. learn-model-evaluation-with-phoebe defines the metrics and owns monitoring for drift.

Session 1 cheat sheet

A score is a betThat the data ahead resembles the data it was scored on, in the ways that matter.
Three linksThe same world, an untouched draw, the right target. They break for different reasons.
Link one is nobody's faultThe world moves for reasons unrelated to the model. That is why it needs a watcher, not a process.
0.949 to 0.765One expired arrangement, no retraining, no error anywhere in the pipeline.
Predictive is not durableA held-out score cannot tell the two apart, because a shortcut works on held-out data too.
More data buys precisionNot validity. A tight estimate of the wrong quantity is still the wrong quantity.
Ask the shuffle questionWhat would this model do if one named feature were scrambled. Silence is the finding.
The sentenceScores X on data from when, holds while the resemblance holds, and this person notices in this long.