The six sessions 🚢 · from the launch plan to the decision to stop
The running data is Elec2, 45,312 half-hours of the New South Wales electricity market from 1996 to 1998 and the most used concept-drift benchmark there is. The frameworks are Andrew Ng's Machine Learning in Production, the Hidden Technical Debt paper, the ML Test Score and Google's Rules of Machine Learning, each read at source.
The first deployment is the middle
The lifecycle as a loop with the launch in the middle, the small box of ML code inside a production system and the debts around it, and a readiness score that takes the minimum. Build a one-page readiness sheet.
📐 Session 2 · advancedScope it like a project
Diligence on feasibility and on value before any data is touched, the baseline a model has to beat, metrics first, and the first rule of machine learning: do not be afraid to launch without it.
🐤 Session 3 · advancedShadow, canary, or all at once
Degrees of automation from human only to full automation, choosing between shadow mode, canary and blue-green, and writing the rollback before the rollout. Why training and serving skew is the test teams skip.
🎛️ Session 4 · bench nightThe retrain-policy bench
15,104 real Elec2 rows replayed live under eight policies. Weekly refits buy 4.85 points over never for 130 retrains; an alarm buys 4.08 for 6; retraining before every prediction is worse than no model at all.
📟 Session 5 · advancedMonitor what you will act on
Software, input and output signals, what to watch while labels are late, an action wired to every alert, and the feedback loops that let a model grade its own homework.
🛑 Session 6 · advancedWhen not to productionize
What the famous failure rates really say, the kill criteria you write before launch, and the capstone: a one-page production decision memo that can say ship, hold, or stop.
Your passport 🎫
Clear all three quiz questions in a session and it gets stamped. Six stamps means you can scope an ML project against a baseline, choose how it goes out, set a retrain policy with its price on it, wire monitoring to actions, and write the memo that decides whether it runs at all. Stamps live in this browser only.
What this is not, so you land in the right place 🧭
| If your question is | Go here instead |
|---|---|
| How do I track experiments, register models, and write the deploy pipeline and CI? | DataOps, builder sessions b5 and b6 ↗ |
| Which drift statistic do I compute, and how do I run evaluation continuously? | Model Evaluation, session b9 ↗ |
| Who validates and signs off the model, and what does ongoing monitoring governance require? | Model Risk ↗ |
| How do I monitor pipeline freshness, volume and schema before the data reaches the model? | Data Observability ↗ |
| My model is wrong in places. What do I fix next? | ML Strategy ↗ |
| I have to decide whether, how and for how long a model runs in production, and defend the decision. | you are in the right place |
The decisions, not the plumbing. Every session ends in a document a leader signs: a readiness sheet, a scoping memo, a rollout plan, a retrain policy with a price, an alert-to-action table, a production decision memo. The tools that carry them out are taught on the neighbouring shelves and linked from each page.
Choose your path 🗺️
Three doors into the same conviction: shipping a model is a decision you keep making, and every one of those decisions has a price you can put on the page.
What is honest here 🧾
The bench runs on real public rows, recomputed in your browser. Frameworks are quoted from their primary sources. Where a claim comes second-hand, the page says so, and where a famous number has no source, the page says that instead.
| Claim | Status |
|---|---|
| Every accuracy figure and retrain count on the bench | Really computed in your browser from 15,104 rows of Elec2 (Harries 1999, OpenML dataset 151), and matched row for row by an independent Python run |
| The dollar cost of one retrain | Set by you. $500 by default, a placeholder for validation, review, rollout and on-call time; the accuracy numbers never depend on it |
| Sculley et al. 2015, the ML Test Score 2017, the Rules of Machine Learning | Read at source. The "5 percent ML code" line is quoted with its hedge, because the famous figure itself carries no number |
| Andrew Ng's lifecycle, deployment patterns and degrees of automation | Course pages read; lecture content reported from compiled lecture notes, because the official notes sit behind the course login. The videos and the certificate stay official |
| Chip Huyen on distribution shift, feedback loops and retraining | Read in the author's own blog posts; the book itself was not opened and none of its numbers are quoted |
| "87 percent of models never reach production", "85 percent of AI projects fail" | Traced and corrected. The first comes from a sponsored panel write-up with no survey behind it; the second is a 2018 forecast about bias, not a failure rate |
| Worked examples: the readiness scores, the scoping memo, the rollout plan, the alert table, the decision memo | Constructed for teaching, and labelled so on each page |
The knowledge map 🧠
The whole cut at a glance - hover a session to spotlight its concepts, click any node to jump in.