What would change your mind

"The model performs well" survives every observation. So does "it may degrade over time", and so does "we will keep an eye on it". Nothing that could happen next quarter is inconsistent with any of them, which means none of them is doing work. A claim with no falsifier is not a weak claim; it is not a claim, because there is no state of the world it rules out.

The repair is small and it is entirely a matter of when. Written after the drop, a threshold is a negotiation about whether the drop counts. Written at approval, the same threshold is an agreement, and the only thing left to do in March is read a number. The sentence costs a minute in the meeting where everyone is calm and saves the argument in the meeting where nobody is.

One camp treats the falsifier as a logical test: the observation arrives, the claim is refuted, the model comes out. The other camp points out that no single observation ever refutes a model claim on its own, because a bad number can always be attributed to the sample, the threshold, an implementation change or an unusual month. On that reading a falsifier is not a test at all; it is a commitment device, useful because it removes the option of choosing the explanation after seeing the number.

The disagreement is real and it is not settled. What both sides accept is the part that matters for a review: the threshold has to be written before the data arrives, because whoever writes it afterwards is choosing the verdict. Pick whichever justification you find convincing and write the same sentence.

• Warrant. The reason a claim is believable. A score plus the assumptions that let it travel.
• Underspecification. A pipeline that can return many models with equally strong held-out performance which behave differently in deployment. Session 2.
• Optimism. The gap between a number reported after selecting on a set and the same model's score on data nobody selected on. Session 3.
• Construct validity. Whether an instrument reads the quantity you care about. Session 4.
• Leakage. Information about the outcome reaching the inputs. Session 5, at the scale of a literature.
• Falsifier. The observation that would end the claim, named in advance with a metric, a population, a threshold and a date.

The seed question costs three retrains. At Thornbury, twenty models that differed only in their random seed sat within a held-out spread of 0.003 to 0.019 depending on the data draw, and pulled apart by 0.034 to 0.051 once the wards moved: the shifted spread ran 2.7 to 11.3 times the held-out spread, in five draws out of five. Three retrains will not reproduce that whole picture, and they do not need to. They tell you whether the number in front of you identifies one model or a family.

The fresh-set question costs one scoring run. The measured optimism at Thornbury from selecting the best of a hundred candidates was 0.0072, and the same model's accuracy on data nobody selected on barely moved between one candidate and a hundred: 0.9500 down to 0.9497. That is small, and the point of asking is not to catch a scandal. It is that the answer takes an afternoon, so an unanswered version of it is a choice rather than a constraint.

A published rank and a published paper both sit outside your control. You cannot rerun somebody else's benchmark on your population, and you cannot re-split their data. When the answer that would change your mind is unobtainable, the finding is not that the evidence is bad; the finding is how much weight it can carry, which is less than a number of its size suggests.

What you can do is bound the exposure. Kapoor and Narayanan found leakage in 17 fields, collectively affecting 294 papers, with a taxonomy of eight types. That base rate is an argument for asking how much of a decision rests on evidence nobody in the room produced, and for funding the one cheap replication that would move the answer, rather than for discounting the literature wholesale.

The most common objection to writing a number is that nobody knows the right one. That is true and it is not a reason to leave it out, because a threshold is a statement about what the organisation would do, not an estimate of anything. It sits between the score that justified approval and the score at which the model is doing harm, and both ends of that range are usually known or arguable in the room.

At Thornbury the approved figure was 0.949 and the post-reorganisation figure turned out to be 0.765. A threshold anywhere in between converts the fall into a scheduled decision rather than a discovery. Picking it badly is recoverable; the sentence can be rewritten at the next review with the reason for the change recorded. Picking nothing is what leaves the question open for eighteen months.

Set so low it never fires. A threshold chosen to be safe from embarrassment is a threshold chosen to be inert. The test is simple: ask what the metric would have to read for the sentence to trigger, then ask whether the model would still be usable at that level. If the honest answer is no, the sentence is protecting the approval rather than the patients.

Attached to a metric nobody computes. A falsifier written against a quantity that requires a special study will not be evaluated, whatever the note says. Pick something already on a dashboard or already produced monthly, even if it is a rougher measure, because a rough check that happens beats a precise check that does not.