Where we start
Stopping is the decision organisations are worst at, because a model in production has a team, a dashboard and a line in someone's objectives, and switching it off reads as failure. It is not. Session 4's bench showed a real case where the right production answer was a one-line rule, and another where the most diligent-looking policy was worse than having no model. A team that can say "stop" with evidence is a team whose "ship" means something.
★ What you walk out with tonight. The true source of three failure-rate numbers you will hear again, four questions that decide ship, hold or stop, a retirement sequence that does not break the systems that quietly depend on the model, and a signed one-page production decision memo for one real model.
Three famous numbers, traced
5 min liveKill decisions are often argued with a statistic: "most ML projects never reach production anyway." Three versions travel. Each was read at its source for this session, and only one of them is a survey result.
LiveWhy the folklore matters for a kill decision3 min▶
A failure rate cuts both ways in a meeting. "Most projects fail" is used to excuse a project that is failing and to dismiss one that is working. Neither use is honest, because none of the three numbers says anything about your model. The 87 percent comes from a sponsored conference write-up in which two speakers, from IBM and Gap, discussed why projects stall; the article's own sentence says "13%" and "one out of every 10" in the same breath and cites no study. The 85 percent is a 2018 Gartner prediction about biased outcomes, retold as a failure rate. The 53 percent is a real survey average, from Gartner's 2019 AI in Organizations survey, and a later survey put it at 54 percent; it is what respondents reported about their own organisations.
The professional norm is simple: when a number enters a production decision, name its source and what kind of evidence it is. For a kill decision, the only evidence that counts is the model's own: its gap over the baseline, what it costs to run, and what happens when it is off.
Self-studyWhat the 53 percent does and does not tell you2 min▶
A survey average says that, among respondents, roughly half of AI projects were reported as reaching production. It does not say the other half should have, and it does not say the half that shipped were worth running. A project stopped at the scoping memo of session 2 counts as "did not reach production" and may be the best decision in the portfolio. Read the number as a base rate for planning, never as a verdict on a single project.
A sibling page got here first. The same two figures, the 85 percent forecast and the 87 percent claim, are treated as not stated as fact on the revenue-model session of the Ship Playbook shelf. When two independent traces reach the same verdict, that is the moment to stop quoting a number, not the moment to find a third source for it.
Four questions decide ship, hold or stop
6 min liveEvery question below was built in an earlier session, and each has a number behind it. The order matters: the first one can end the discussion before any of the others is asked, and it is the one teams skip.
LiveQuestion 1 is Rule 1, measured2 min▶
Zinkevich's first rule, "Don't be afraid to launch a product without machine learning", becomes a test once the no-model rule has a number. On the Elec2 bench with labels 1.5 hours late, repeating the last label scores 74.63 percent and the best model policy, weekly refits on all history, scores 73.51. That model fails question 1 and the tree stops: ship the rule. With labels a day late the rule falls to 65.35 and the same model, at 73.55, passes by 8.20 points. The same model, the same market: the label delay decided it.
LiveQuestion 2 puts a price on the gap2 min▶
A model that clears the baseline still has to pay its way. The bench prices the retraining part: at $500 a retrain, a figure you set, weekly refits cost $65,000 over the replay and the alarm policy $3,000 for most of the same accuracy. Add what session 5 wired up, the monitoring and the on-call behind every alert, and what session 1 counted, the nine boxes around the model. Then ask the business owner what one point of accuracy is worth in their unit, calls, pounds, hours. If the answer times the gap is smaller than the running cost, the honest exit is stop, however good the model looks on its own.
Self-studyQuestions 3 and 4 hold, they do not kill2 min▶
A readiness score of 0, "more of a research project than a productionized system" in Breck et al.'s words, or a box nobody owns, is a reason to hold and pay the debt, not a reason to cancel a model that is worth running. The same goes for a rollback that has never been rehearsed or a fallback no one can run: fix it, then ship. Keeping "hold" separate from "stop" stops teams from killing good models for fixable reasons and from shipping bad ones because the infrastructure happens to be ready.
Stopping is a deployment too
4 min liveA model that is switched off in one step breaks whatever quietly depended on it. Sculley et al. call those systems undeclared consumers: systems "silently using the output of a given model as an input to another system". Retirement runs the rollout of session 3 in reverse, with the fallback as the new version.
LiveWhy the note at the old endpoint matters2 min▶
A model that vanishes leaves questions: was it broken, was it replaced, can it come back? A short note where it used to live, with the date, the decision memo and where the archive is, answers them for the next team, which may be yours in a year. Sculley et al.'s warning about dead experimental codepaths applies here too: code nobody runs any more should be deleted, and a retired model's serving path is exactly that code.
Self-studyStop is also a review outcome, not only a launch outcome2 min▶
Session 4's bench shows why a model that passed the tree at launch can fail it later. A never-retrained Elec2 model scores 68.70 percent over the replay against 65.35 for the no-model rule at a one-day delay, but its accuracy sinks into the low 60s for long stretches, blocks where the rule is ahead. Put a review date on every "ship", re-run the four questions on it with the current numbers, and let the answer be stop. Formal validation and sign-off of that review, where a regulator requires it, belongs to Model Risk ↗.
The electricity benchmark's own kill decision. Žliobaitė's 2013 note on Elec2 is, in effect, a stop memo for a research habit: many adaptive methods published on the dataset did not beat repeating the current label, 85.3 percent on the full file, and a detector firing at random improved accuracy. Her recommendation, compare every result against that rule, is question 1 of the tree, written in 2013.
Capstone: the one-page production decision memo
★ 22 min · everyone buildsOne page, six blocks, one decision at the top. It pulls a number from every earlier session, which is why it is the capstone: if a block cannot be filled, the session that owns it is where to go back. Use a real model you own or approve. If you have none, use the Elec2 price-direction model; a constructed example sits after the steps.
Write the decision first. One word and a sentence, signed and dated. Writing it before the evidence feels backwards; it forces the evidence blocks to argue for or against something specific.
The gap over the no-model rule, at the real label delay. This is question 1 of the tree. If the gap is zero or negative, the decision is stop and the rest of the memo describes the retirement.
The running cost against the value of the gap. Retrains priced the way session 4 priced them, plus monitoring and on-call from session 5. Value per point comes from the business owner, in their unit.
Readiness and safety. The minimum section score from session 1, the boxes without owners, and the rollback and fallback from session 3. Any gap here is hold, not stop.
The review date and the stop rule. Every ship gets a date on which the four questions are asked again with current numbers, and a rule written now that would make the answer stop.
Northline Grid, a constructed electricity retailer, real bench numbers. Northline predicts the NSW half-hour price direction to time its hedges. Its settlement feed gives it the true direction a day late. Block 1: the no-model rule scores 65.35 percent, the weekly-refit model 73.55, a gap of 8.20 points. Block 2: 130 retrains at $500 is $65,000; the alarm policy gets 72.78 percent for $3,000, so the memo switches the policy to the alarm and keeps a 7.43-point gap. Decision: ship, review in a quarter. Then the data team negotiates a feed that delivers labels within 1.5 hours. At that delay the rule scores 74.63 and the best model policy 73.51. The same memo, re-run on review day, says stop: ship the rule. Northline, the hedging use and its costs are constructed; every accuracy figure is what the session 4 bench prints for that setting.
Put the four questions in code next to the model, fed by the numbers the replay harness from session 4 and the readiness file from session 1 already produce, so the review day re-runs the tree instead of re-arguing it.
Scheduling the review job and wiring it to the registry is tooling; DataOps session b6 ↗ builds it. If the memo is going to a board, the one-page structure is the same one ML Strategy a6 ↗ uses for what to fix next.
After the series
30 min- Finish the memo and get both signatures. The business owner signs block 2's value per point; the model owner signs everything else. A memo with one signature is a draft.
- Run the four questions on a model already in production that nobody has reviewed in a year. Write down which question you could not answer; that is the finding.
- Replace one failure-rate statistic in a strategy deck you own with the model's own evidence, or with the survey figure quoted as a survey.
- Pick one model to retire, even hypothetically, and list its consumers. If you cannot list them, block 1 of the retirement sequence is your next piece of work.
Three questions
Clear all three and session 6 gets stamped in your passport.
1 · A strategy deck says "87 percent of ML projects never reach production (VentureBeat)". What is accurate?
A describes the 2021 Gartner survey behind the 54 percent, B the 2018 Gartner forecast behind the "85 percent fail" retelling, and C the ML Test Score paper, which measured test adoption. Only D is where the 87 comes from.
2 · A model beats the no-model rule by 8.20 points, but its readiness score is 0 because nothing in monitoring is done. What does the kill tree say?
Questions 3 and 4 hold, they do not kill. Keeping hold separate from stop prevents cancelling a model worth running for a fixable reason, and prevents shipping one with no monitoring. Retraining does not add monitoring.
3 · Why does retirement start with finding the model's consumers?
Retirement is a deployment of the fallback, run in stages like session 3's rollout. The consumers are the part of the system most likely to be unknown, which is why they come first.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 6 cheat sheet · pin this
That is the whole cut: a readiness sheet, a scoping memo, a rollout plan, a retrain policy with a price, an alert-to-action table and a signed decision memo. Take the memo to the next review of a model you own. The tooling that carries these decisions out lives on the neighbouring shelves: DataOps for the pipelines, Model Evaluation for drift statistics, Model Risk for validation. Back to the course home for the knowledge map.