The assumption under the split

Hold out a random slice of your rows, train on the rest, score on the slice. What you have built is an unbiased estimate of the model's performance on the distribution the slice was drawn from. That is a real guarantee and it is worth having. It is also the whole guarantee. The estimate is not slightly informative about other distributions, or weakly informative, or informative on average: it is silent about them, because nothing in the procedure ever touched them.

The assumption doing the work has a name. The rows are treated as independent and identically distributed: each one an independent draw from one fixed distribution, so a random half of them stands in for the rest. Both halves of that phrase are claims about how the data came to exist. Independent says one row does not carry information about another. Identically distributed says the world did not move while you were collecting. Neither is something the split function can check, and neither one is repaired by drawing the slice more carefully.

• Independent and identically distributed. Each row an independent draw from one fixed distribution. The assumption a random split rests on, and a statement about the world rather than about the code.
• Held-out set. Rows kept out of training and used to estimate behaviour on unseen data from the same distribution.
• Group. The unit that repeats across rows: a patient, a session, a device, an institution. If rows share a group, they are not independent draws.
• Group-aware split. A split where every row belonging to one group lands on the same side, so the score answers a question about a new group.
• Time-based split. Train on the earlier period, score on the later one, so the model is never shown the future of the period it is judged on.
• Stress set. Rows chosen to break one named assumption, scored alongside the held-out set rather than instead of it. Session 4.
• Shortcut. A feature that predicts the outcome without causing it, whose usefulness depends on an arrangement that can change. Session 2 builds one and measures the model's reliance on it.
• Accuracy and the rest of the metric family. Defined, computed and monitored in learn-model-evaluation-with-phoebe; here a metric is only ever a claim whose warrant is in question.

This is the sentence the whole track hangs on, and it is short. A shortcut is a feature that predicts the outcome without causing it. In the collection year, Thornbury's ward agrees with the outcome about 90 percent of the time. That agreement is a property of the world the data was collected from, which means it is a property of the training rows and of the held-out rows in equal measure. A model that leans on the ward scores well on both. A model that ignores it scores worse on both, and would be rejected in review for being worse.

So the held-out set cannot distinguish the two kinds of learning, not because it is too small and not because it was built carelessly, but because it was drawn from the same place. No amount of additional held-out data changes this. The only measurements that can separate them are measurements that do something the split does not do: intervene on a feature and watch the predictions move, which is session 2, or score the model on data drawn from somewhere else, which is session 4.

Keep the language plain. The vitals drift and the lab trend at Thornbury are written into the generator as things that actually bring the outcome about. The ward is written in as something that merely agrees with the outcome, because in the collection year the high-acuity ward received most of the patients who went on to deteriorate. Both kinds of feature raise a held-out score. Only one of them keeps working when the arrangement behind the agreement is changed by somebody in facilities management.

That distinction is all that is needed tonight, and the formal machinery for reasoning about it, identification and what you may and may not adjust for, belongs to learn-causal-inference-with-phoebe. What is needed here is the habit of asking which kind a feature is, and the knowledge that the held-out score will never answer.

Unbiasedness is a statement about repetition: run the whole procedure many times on fresh draws from the same distribution and the estimates centre on the true value for that distribution. It says nothing about how far any single estimate sits from the truth, which is variance, and nothing about whether the distribution is the one you care about, which is validity. A large held-out set buys the first. Nothing you can do inside the collection year buys the second.

The practical reading is that precision and relevance are bought in different shops. When somebody proposes quadrupling the held-out set to increase confidence before deployment, they are buying precision on a quantity nobody disputed.

Rows are rarely the unit anybody cares about. Admissions belong to patients, clicks belong to sessions, readings belong to devices, cases belong to institutions. When a random split scatters the rows of one group across both sides, the held-out score is still an unbiased estimate, but of a strange quantity: performance on a new row from a group the model has already met. If deployment means meeting a new patient, that quantity is not the one you wanted, and it is reliably the friendlier of the two.

The repair is a group-aware split, and the only hard part is naming the group honestly. It is usually the unit that will be new at deployment time. Write it down before the split, because after the split it is invisible: the two index sets look identical in shape and completely different in meaning.

When rows carry a clock, a random split takes some rows from after the scored period and puts them in training. The model then has information about how the period turned out while being graded on it. The estimate that comes back is unbiased for a world in which you get to know the future, which is not a world anybody deploys into.

The repair is a time-based split: train on the earlier period, score on the later one, and accept a smaller and noisier number in exchange for one that means something. This is a different failure from a field that could not have been known when the prediction was made, which is leakage; the mechanics of that, how a post-outcome field reaches a feature list and how a noisy one survives review, belong to learn-model-risk-with-phoebe p3.

The third shape is the one Thornbury eventually meets. Collection happens in one period; deployment happens in another; between them the people change, or the process that produces the rows changes, or an arrangement that several features quietly depended on is reorganised. Nothing inside the collection window can measure this, because every row you could add is drawn from the window.

A stress set is data chosen to break one named assumption: a later period, a different site, a rearranged version of the world. It is scored alongside the held-out set, never instead of it, and it answers a narrow question rather than a general one. Designing one, and being precise about what it can and cannot establish, is session 4.