Leader track 🧾 · six sessions, no code
For whoever approves the deployment. What a score claims, why two equally good models are not the same model, what a benchmark licenses, how to treat evidence you did not produce, and the question to ask before your name goes on it.
What a test score entitles you to
A held-out number is a claim about a future that resembles the past. Three assumptions carry it, and one of them is nobody's fault when it breaks.
🎲 Leader session 2Equally good, differently wrong
Twenty models, one pipeline, identical where you looked and three to eleven times more spread where you did not.
🎯 Leader session 3When the number is a target
What selecting against a test set does to it, measured, against the field evidence that says it matters less than feared.
🏆 Leader session 4What a benchmark licenses
Construct validity and the benchmark lottery: a rank is a property of a pairing, not of a method.
📚 Leader session 5A whole literature, wrong the same way
294 papers across 17 fields, and what to do about evidence you did not produce.
🔄 Leader session 6What would change your mind
The falsifier, written before the result. A claim with no falsifier is not a weak claim, it is not a claim.
Practitioner track ⚖️ · ten sessions, hands on
For the people who produce the evidence. The split, the shortcut, the seed sweep, stress tests, selection, test-set size, benchmarks, leakage at field scale, a finding that died on its second data draw, and a capstone bench.
The assumption under the split
What a random split buys, and the three deployments where it is a lie about the world.
🔀 Practitioner session 2The shortcut
A feature that predicts without causing. Flip it on every patient and count how many predictions move.
🌱 Practitioner session 3Twenty seeds, one pipeline
Change nothing but the seed. Held-out spread 0.003 to 0.019; shifted spread 0.034 to 0.051.
🧪 Practitioner session 4Stress tests
Designing the shifted set, and what a stress test can and cannot establish.
🔎 Practitioner session 5Selecting on a set
The reported number climbs to 0.9569 while the honest number sits at 0.9497. The gap is the optimism.
📏 Practitioner session 6How big does a test set need to be
Selection optimism 0.0091 at n=200 and 0.0030 at n=2000. At n=100 the estimate is too noisy to rank itself.
📖 Practitioner session 7Reading a benchmark
What transfers from a published ranking to your problem, which is less than it appears.
🕳️ Practitioner session 8Leakage at the scale of a field
Eight types, 17 disciplines, 294 papers. A structural failure, not a careless one.
💥 Practitioner session 9The finding that did not survive
A correlation of -0.89 that looked like the headline, until the fourth data draw returned +0.09.
⚖️ Practitioner session 10The credibility bench
Both benches live. Train them yourself, then write the falsifier before you look.
The whole thing on one page
Every idea in the course and the session that owns it. Click any node to jump there.
Three ways through
Everyone starts at session 1 of their track. These are the shortest useful routes if you cannot take all sixteen.
What is real here
Both benches train networks when you click. Nothing is a stored result.
| The patients | Generated from relationships written out in assets/ep-live.js: two signals cause deterioration, one ward does not |
| Every model | Three inputs, twelve tanh hidden units, trained by stochastic gradient descent in the page |
| Every accuracy | Counted on data the model did not train on. Nothing is hard-coded |
| The finding that died | A correlation of -0.89 looked like the headline until the fourth draw returned +0.09. Session 9 keeps the record |