Where we start
Every session in this series is a decision a leader signs and a team then lives with: what to scope, how to roll out, when to retrain, what to watch, when to stop. None of them is a tooling choice, and all of them get made by default when nobody makes them on purpose. This first session sets out why they exist at all. A model in production is not a finished artifact; it is the smallest part of a system that has to keep working while the world it predicts moves underneath it.
★ What you walk out with tonight. A one-page readiness sheet for one real ML system: its boxes and their owners, its ML Test Score by section with the minimum taken honestly, and the three debts you will pay first. Plus the sentence that reframes every launch review: "the first deployment is the middle of this project, not the end."
The lifecycle is a loop with a launch in the middle
5 min liveAndrew Ng's Machine Learning in Production course lays the project out in four stages: scoping, data, modeling, deployment. Drawn as a line it looks like a plan with an end. Drawn the way it actually runs, three of the arrows point backwards, and the launch sits in the middle of the timeline, not at its right edge.
LiveWhat each stage decides, and who should sign it3 min▶
Scoping decides what to work on and what success means, in business terms, before any data is touched. Session 2. Data decides what the inputs and labels are, how they are defined consistently, and what baseline the model has to beat. Modeling trains, then does error analysis to decide whether the next gain comes from the model or the data. Deployment is where the system meets real traffic: how it rolls out (session 3), how it stays fresh (session 4), what is watched (session 5).
The leader's job is at the two ends. Scoping is the only stage where "no" is still cheap. Deployment is the stage where the recurring bill starts, and it is the one most plans budget as a single line item.
Self-studyThe proof-of-concept to production gap3 min▶
Ng's lectures name the distance between "it works in the notebook" and "it works for customers" the proof of concept to production gap (reported, from compiled lecture notes; the official notes sit behind the course login). The same lectures say that in many projects maybe only 5 to 10 percent of the work is the machine learning code itself. Treat that as a practitioner's estimate, not a measurement: it is a different figure from the one in Part 2, and neither paper nor lecture measured it.
What closes the gap is not a better model. It is the work the next five sessions are about: a scoped problem with a baseline, a rollout that can be reversed, a retrain policy somebody chose, monitoring wired to actions, and a standing willingness to switch it off.
A model that was "up" the whole time. Session 4 replays two and a half years of a real electricity market, Elec2, against a model trained on its first four weeks and never touched again. Nothing crashed. Every prediction was served on time. And on the same rows, a policy that refits once a quarter is 3.75 accuracy points ahead of it, because the market the model learned no longer existed. A system can pass every uptime check and still be quietly wrong, which is the whole reason the right half of the timeline exists.
The model is the small box
6 min liveIn 2015 ten Google engineers led by D. Sculley published "Hidden Technical Debt in Machine Learning Systems" at NeurIPS. Its first figure is the most reproduced picture in production ML: a tiny black box labelled ML Code, surrounded by nine much larger boxes of everything else a real system needs. The caption says only a small fraction of real-world ML systems is the ML code. Here it is redrawn, with the one thing everyone budgets for in ochre.
LiveRead the percent carefully, because everyone quotes it2 min▶
The figure is often retold as "ML code is 5 percent of an ML system". The figure itself carries no percent. The number appears later, in the section on glue code, hedged: a mature system "might end up being (at most) 5% machine learning code and (at least) 95% glue code", written as an argument for building native solutions rather than wrapping general-purpose packages. Ng's lectures offer "maybe only 5-10%" as a separate estimate. Quote either with its hedge and its author, or quote the caption, which is the claim the paper actually makes: a small fraction.
LiveThe debts a leader should be able to name4 min▶
The paper's catalogue is long. Six of its debts show up in almost every production review, and each one is a decision, not a code smell.
| Debt | What it means | The decision it forces |
|---|---|---|
| Entanglement, CACE | "Changing Anything Changes Everything": no input's effect is independent of the others | Every change, even one feature, is re-validated as a new model |
| Undeclared consumers | Other systems silently use the model's output as their input | Who is allowed to read the output, and who is told when it changes |
| Unstable data dependencies | An input signal changes behaviour over time, often owned by another team | Version the signals you depend on; give each an owner |
| Feedback loops | The model influences the data it will later be trained on, directly or through the world | Session 5: what you hold out so the model cannot grade itself |
| Glue code, pipeline jungles | Most of the code moves data in and out; joins and scrapes pile up | Budget the plumbing as the product, not overhead |
| Changes in the external world | Fixed thresholds go stale; the world moves under the model | Sessions 4 and 5: retrain policy and monitoring you will act on |
Self-studyThe rest of the catalogue3 min▶
Correction cascades (a model patched on top of another model, until improving the first breaks the second: the paper calls it an improvement deadlock). Underutilized data dependencies (legacy, bundled, epsilon and correlated features that add little and break often; the paper suggests leave-one-feature-out evaluations to find them). Dead experimental codepaths. Abstraction debt. Configuration debt ("the number of lines of configuration can far exceed the number of lines of the traditional code"). And four debts outside the code: data testing, reproducibility, process management and cultural debt.
The paper's own warning about dead code. Sculley and colleagues cite the Knight Capital incident, in which a firm lost $465 million in 45 minutes, apparently because obsolete experimental codepaths behaved unexpectedly. It is not an ML failure, and the paper does not claim it is. It is the reason the paper tells ML teams to delete the experiment branches they no longer run: in a system where most code is not the model, the code nobody remembers is the code that surprises you.
Readiness is scored by the weakest section
4 min liveTwo years after the debt paper, Eric Breck, D. Sculley and three colleagues turned the warning into a checklist: 28 tests in four sections of seven, for data, model, infrastructure and monitoring. The scoring rule is the part worth stealing. Each test earns half a point if it is run by hand and documented, a full point if it runs automatically. The final score is not the total. It is the minimum of the four section scores.
LiveWhy the minimum is the leader's rule2 min▶
A total would let a team with superb infrastructure and no monitoring report a healthy number. The authors take the minimum because "all four sections are important, and so a system must consider all in order to raise the score". That is a governance rule, not a statistics choice: it tells a team that the next point of readiness can only come from its weakest area, which is exactly where the next unit of effort should go.
| Final score | What the paper says it means |
|---|---|
| 0 | More of a research project than a productionized system |
| (0, 1] | Not totally untested, but it is worth considering the possibility of serious holes in reliability |
| (1, 2] | There's been first pass at basic productionization, but additional investment may be needed |
| (2, 3] | Reasonably tested, but it's possible that more of those tests and procedures may be automated |
| (3, 5] | Strong levels of automated testing and monitoring, appropriate for mission-critical systems |
| above 5 | Exceptional levels of automated testing and monitoring |
Self-studyWhat the 36 Google teams showed2 min▶
The authors met 36 teams across Google. None of the 28 tests was implemented by more than 80 percent of them. The one they single out, "perhaps the most important and least implemented", is monitoring test 3: training and serving are not skewed, meaning the features the model sees live are computed the same way as the features it was trained on. Session 3 comes back to it, because a shadow deployment is the cheapest way to catch that skew before customers do.
A one-page readiness sheet for a system you own
★ 22 min · everyone buildsPick one ML system: one you run, one you are about to approve, or one you depend on. If you have none, use the Elec2 price-direction model from session 4. Five steps, each a block you copy into a document and fill in. There are no right answers to type; there are honest ones and optimistic ones, and the sheet is only useful if it is the first kind.
Name the decision, not the model. One sentence each. The fallback line matters most: it is what the business does when the model is switched off, and if nobody can write it, the model has no safe "off".
Draw your nine boxes and put a name on each. Use Sculley's list as the checklist. A team name is not an owner; a person who gets paged is.
Score the four sections. Seven tests each. Zero if it is not done, 0.5 if someone does it by hand and writes the result down, 1 if it runs automatically. Monitoring is below; the other three sections are in the paper, and session 3 and 5 come back to the ones leaders ask about.
Take the minimum and read the band. Write the lowest section score as the system's score. Resist the urge to average. Then copy the band sentence from Part 3 next to it, word for word.
Pick three debts to pay first. From the lowest section, or from any box without an owner. Each gets a person and a date. That list, not the score, is what the sheet is for.
Keep the sheet as data in the model's repository, not as a slide, so the score is recomputed on every review and the history shows whether it moved. A few lines are enough.
Those scores are the constructed example from the Part 3 figure. The automation that turns a 0.5 into a 1, scheduled checks, alerts and pipeline tests, is built in DataOps session b6 ↗.
Date the sheet. readiness_2026-09-29.md, not readiness.md. Session 6 asks whether this system should still be running, and the answer is much easier when you can show the score it had when it launched and the score it has now.
Before session 2
30 min- Finish the sheet with the people who run the system, not alone. The first disagreement about a score is usually the most useful line on the page.
- Find one undeclared consumer. Ask the data platform team who reads the model's output table. If the list surprises the model's owner, write it into block 2.
- Write the fallback in one sentence and ask the business owner whether it is true. If the honest fallback is "we would not notice", that is a finding for session 5.
- Bring one project that has not launched yet. Session 2 scopes it before a line of model code exists, including the version of it that uses no machine learning at all.
Three questions
Clear all three and session 1 gets stamped in your passport.
1 · A board slide says "Google showed ML code is only 5% of an ML system." What is the accurate version?
The figure has no numbers. The hedged 5 percent is in the text, and the 36 teams belong to a different paper, the ML Test Score, which measured test adoption, not code share.
2 · A system scores 6 of 7 on infrastructure, 5 on data, 4 on model development and 0.5 on monitoring. What is its ML Test Score?
The final score is the minimum of the four section scores, so strength elsewhere cannot hide the weakest section. The next point of readiness can only come from monitoring.
3 · Why does the lifecycle diagram put the first deployment in the middle of the timeline?
The three arrows back are the point: deployment feeds data and modeling again. "The middle" is a claim about where the work continues, not a measured ratio of durations.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 1 cheat sheet · pin this
The cheapest moment to stop a bad ML project is before it starts. Session 2 scopes a project the way Ng teaches it, with diligence on feasibility and on value, a baseline the model must beat, and Zinkevich's first rule of machine learning: do not be afraid to launch without it. Bring the project that has not launched yet.