Monitor what you will act on

Andrew Ng's Machine Learning in Production lectures teach monitoring as a brainstorm before it is a dashboard (reported, from compiled lecture notes). Sit down with the team and list everything that could go wrong. Then pick metrics in three families: software metrics such as memory, compute, latency, throughput and server load; input metrics that describe the distribution of what the model is fed; and output metrics that describe what the model returns. Set a threshold on each, and expect to adapt both the metrics and the thresholds over time as you learn which ones move when something real happens.

The last step is the one most teams skip. A threshold set on launch day is a guess. Treating it as permanent is how a channel fills with alerts nobody reads, and Part 2 is about pricing that.

Breck and colleagues give monitoring its own section of seven tests, scored like the other three: half a point by hand and documented, a full point automated. Session 1's constructed readiness sheet scored this section lowest, at 1.0 of 7, and that is the shape to check your own sheet against; the paper itself does not rank the sections by how often teams fail them.

Six of seven can alarm on the day something breaks. Only the seventh measures the thing the business pays for, and it is the one that runs a label delay behind.

Chip Huyen's blog post on data distribution shifts gives the cleanest definitions (read, blog). Covariate shift: P(X) changes while P(Y|X) stays the same; the inputs move, the rule linking them to the outcome does not. Label shift: P(Y) changes while P(X|Y) stays the same; the mix of outcomes moves. Concept drift: P(Y|X) changes while P(X) stays the same; the same inputs now mean a different outcome.

Ng's lectures use a simpler pair (reported): data drift when the distribution of x changes, concept drift when the mapping from x to y changes. He uses the two words loosely at times, so when a team says "drift", ask which one they mean.

The practical consequence is in the figure. An input monitor sees covariate shift, because it is by definition a change in the inputs. It cannot see concept drift, because the inputs have not moved: the only place concept drift shows up is performance against labels, after the labels land. Label shift sits between them; the output distribution may hint at it, but confirming it takes labels too.

This session decides what to watch and why. How to measure a distribution change, which statistic, which test, which window, is taught in Model Evaluation b9, drift and continuous evaluation ↗. Monitors on the pipeline feeding the model, freshness, volume and schema, are taught in Data Observability a2, the five signals ↗. Both plug into the input row of the figure; neither changes the rule that performance waits for labels.

In the section of the hidden-technical-debt paper on changes in the external world, Sculley and colleagues write: "Comprehensive live monitoring of system behavior in real time combined with automated response is critical for long-term system reliability." Then: "The key question is: what to monitor?" They name three starting points (read).

• Prediction bias. In a system working as intended, the distribution of predicted labels should match the distribution of observed labels. A change there is a cheap, early sign that the world has moved. It is not a sufficient test: a null model that predicts the average label for everything passes it perfectly. Slice it by segment and it becomes much sharper.
• Action limits. Where the system takes actions in the world, bidding, approving, blocking, set limits on those actions as a sanity check, and alert when one is hit. The limit is a floor under a model going wrong fast.
• Up-stream producers. Data flowing in from other teams' systems should be monitored against its own expectations, and those producers' alerts should reach the team that consumes the data. Session 1's unstable data dependencies, made operational.

Note the word "automated response". The paper pairs monitoring with action from its first sentence on the subject.

Session 4's bench already priced this, on real Elec2 rows at a one-day label delay. The alarm policy that refits when four-week accuracy on labelled rows falls 10 points below where it started retrains 6 times and scores 72.78 percent. Now make it a hair trigger: fire on any dip at all, watching only the last day. It retrains 209 times and scores 72.79 percent.

The extra 203 retrains buy 2 correct rows out of 14,656. Every one of those retrains is an alert somebody acted on. A threshold decides how often the team is interrupted and how much compute is spent, long before it decides accuracy.

Reproduce it. Open session 4's bench, keep labels at 1 day and refits on all labelled history, then set Retrain to "on an alarm", Alarm when accuracy falls to "any dip at all", and Alarm watches to "1 day". Compare it with the same alarm at "10 points" and "4 weeks".

Martin Zinkevich's Rules of Machine Learning give this failure its own rule, "Watch for silent failures." (read). His examples are the kind no latency chart will ever catch. A feature column that was populated in 90 percent of examples drops to 60 percent, and the model keeps serving, slightly worse, with nothing crashing. A table behind Google Play went stale for 6 months; refreshing it gave "a boost of 2% in install rate".

Both are row 2 of Part 1's figure: an input-side signal, visible without a single label, that nobody was watching. And both have an obvious action: find the upstream owner and fix the feed, which is exactly the second row of the alert chain above.

The same discipline, applied to data pipelines rather than models, is taught in Data Observability a4, alert or noise ↗: how to count alerts per week, how to tell a noisy check from a broken one, and when to retire a check. The rule carries over unchanged: a check that never leads to an action is costing attention and buying nothing.

The hidden-technical-debt paper separates two kinds (read). A direct feedback loop: "A model may directly influence the selection of its own future training data." That is the figure: the approvals decide the labels. A hidden feedback loop is harder to find, because it runs between systems: "two systems influence each other indirectly through the world." Two models owned by two teams, each changing the behaviour the other learns from, with neither team's monitor built to see the other.

Neither loop crashes anything. Both can make every monitor in Part 1 look better while the model gets worse on the cases it no longer sees.

Chip Huyen names the failure a degenerate feedback loop: "a system's outputs are used to create or process the same system's inputs" (read, blog). Recommenders are the familiar case; lending, fraud and hiring screens are the expensive ones.

She also names the dial that sets how fast any loop can be caught: feedback loop length, the time from serving a prediction to getting feedback on it. It is session 4's label delay under another name. For fraud she gives a dispute window of "a month to three months" (read, blog). Her book, Designing Machine Learning Systems, covers the same ground at more depth; its text was not read for this session, so nothing here is quoted from it.

• A held-out slice. A small random share of cases is decided without the model, by the old rule or at random within safe limits, and its outcomes are labelled like any other. It costs some worse decisions; it buys the only unbiased view of what the model turns away. Its size is a business choice, and the table in the build-along gives it a row.
• Log what the model did not choose. Store the score and the decision for every case, including the declined ones. You cannot learn their outcomes, but you can see when the share being declined moves, which is an output-row signal from Part 1.
• Measure the label delay, as session 4's homework asked. The loop length tells you how far behind the performance row is, and therefore how long a loop can run before any accuracy alarm could notice.