Shadow, canary, or all at once

Ng's lectures call the two middle steps "human in the loop" (reported). They are where most of the value sits for the fewest surprises: the model does the easy, confident volume, and people keep the cases where a wrong answer costs the most.

Two numbers pick the step. The first is the cost of one wrong decision acted on without a person checking it. The second is the share of cases where the model is confident enough to be trusted alone, measured on your data, not assumed. A high error cost and a small confident share point to AI assistance. A moderate cost and a large confident share point to partial automation. Full automation needs a low error cost, or a confident share so large and so well measured that the remaining human review would add nothing.

You can stop at any step. Full automation is not the goal by default; for many decisions partial automation is the finished product, because the human queue is exactly where the expensive errors would have gone. Write the stopping step into the plan now, so that "let's automate the rest" becomes a new decision later rather than a drift.

Each step up is its own launch. Give every move a written criterion before it happens: shadow to assistance when the model agrees with the human on enough cases and its disagreements are reviewed; assistance to partial when the people using it accept its confident answers almost always and the confidence threshold has been set on logged data; partial to full only if the human queue has stopped catching anything the model got wrong. The thresholds are yours to set and they will differ by decision. What must not differ is that each one is written before the move, not argued after it.

A shadow run puts the new model on real inputs, computed by the real serving pipeline, at real load. It proves three things cheaply: whether the features it sees live match the ones it was trained on (training/serving skew, Part 3), how often it agrees with the old model or the human, and whether it meets its latency budget under real traffic. Nobody is affected if it is wrong.

What it cannot prove is the effect of acting on its predictions. Its answers are never used, so the world never reacts to them: no customer changes behaviour, no downstream team works a different queue, no feedback loop starts. If the value of the model depends on what happens after the decision, shadow mode tells you nothing about that value. Only live traffic does, which is what the canary is for.

A canary limits the blast radius. If the new model is wrong in a way shadow could not show, only the small share of traffic routed to it is hurt, and for only as long as the canary runs. It also gives you the first measurement of the model's real effect, on a sample, with the old model serving the rest as a live comparison.

The canary is only as good as the promotion rule written before it starts: the metrics the new model must clear against the old one on the same period, the minimum duration or volume before anyone looks, and the result that sends it back. A canary without that rule becomes a slow full launch, promoted because nothing loud went wrong.

Blue-green runs two complete serving stacks. All traffic goes to one; the switch moves it to the other. The old stack stays warm, so rolling back is one switch, not a redeploy. The price is running two full stacks for as long as you want that option, and the discipline to keep the old stack truly deployable: same feature pipeline, same dependencies, still passing its checks. A blue stack that quietly rotted is not a rollback.

Switching all traffic in one move is not reckless by definition. It is the right call when a wrong prediction is cheap and reversible (a ranking tweak, an internal dashboard, a recommendation a person glances past), when traffic is too thin for a small share to tell you anything in a useful time, and when the old model can be restored quickly by redeploying it. What makes it safe is not the pattern; it is a rollback plan that exists before the switch, a monitor that would notice the problem, and an owner who is on call the day it ships.

The ML Test Score puts rollout and rollback in its infrastructure section. Infra 6 is the canary test. Infra 7 reads: "Models can be quickly and safely rolled back to a previous serving version." Both halves of that sentence are tests a team can fail. "Quickly" means somebody has timed it. "Safely" means the previous version still works with today's feature pipeline and today's dependencies, which you only know if you tried.

A rollback nobody has run is a hope. Rehearse it on a quiet day, time it, and write down who can pull it without asking anyone. In the ML Test Score's scoring, a rollback done by hand and documented earns half a point; one that runs automatically and repeatedly earns the full point.

Monitoring test 3 of the ML Test Score is "training and serving are not skewed": the features the model sees live are computed the same way as the features it was trained on. Among the 36 Google teams the authors met, they call it "Perhaps the most important and least implemented test". It is common because the same feature is usually computed twice, once in a batch pipeline for training and once in the serving path, by different code, sometimes by different teams.

Shadow mode is the cheapest place to catch it. The model is running on live inputs, nobody is acting on its output, and you can recompute the training-pipeline version of every served feature on the same rows and compare. Skew found in shadow costs a fix. Skew found in a full launch costs the decisions made on it. Statistical drift tests on the live inputs belong to Model Evaluation b9 ↗.

Martin Zinkevich's Rules of Machine Learning spend three rules on the problem. Rule 29: save the features as they were at serving time and log them, so the model can be trained on exactly what it saw. Rule 32: re-use code between the training pipeline and the serving path, so there is one computation of each feature, not two. Rule 37, "Measure Training/Serving Skew": compare how the model performs on the data it was trained on, on held-out data, on next-day data and live. A gap between next-day data and live data "probably indicates an engineering error", because those should behave the same if the pipelines agree.

Rules 29 and 32 prevent skew; Rule 37 detects what gets through. A rollout plan should say which of the three you have.