A forecast is never "done"
You shipped the forecast in b9 - refit on a schedule, stored, served through an API. But the moment it goes live, the clock starts: actuals arrive, and the only honest question is whether the forecast is still good. Monitoring is the discipline that answers it. This capstone builds the two checks every production forecast needs - a rolling error tracker and a drift detector - and wires them into a closed loop that decides, automatically, when to retrain or fall back. That loop hands the series right back to the analyst from b1.
Track forecast error as actuals arrive 7 min live
Backtesting in b8 measured error before you shipped. Monitoring measures it after - on live actuals the model never saw. Every day, yesterday's forecast meets yesterday's actual; log the pair, and a rolling error metric (MAPE or MASE over the last N days) becomes a dashboard that answers one question: is the forecast still as good as the backtest promised? When that rolling number climbs past a threshold, something has changed.
LiveData drift vs concept drift3 min▶
When error climbs, ask what changed. Two very different causes, two different fixes:
- Data drift - the input distribution shifts. The series moves to a new level or variance (a viral spike, a pricing change, a new market). The relationship the model learned may still hold; it just needs to see the new data. A refit often fixes it.
- Concept drift - the relationship itself changes. A regime change: the weekly pattern inverts, a new channel rewires demand, the pandemic rewrites everything. Refitting on more of the old data will not save you; you may need a new model or new features.
The two look identical on the error chart - both just make MAPE rise. That is why you monitor the inputs too, not only the error. A drift detector on the actuals tells you the distribution moved (data drift); persistent error even after a refit points to concept drift. The M-competition humility applies here as well: when in doubt, a robust simple model degrades more gracefully than a fragile complex one.
Self-studyGuardrails: sanity bounds and the baseline fallback2 min read▶
Monitoring catches slow degradation. Guardrails catch a model that has gone absurd right now, before the number reaches anyone:
- Sanity bounds. A forecast of negative sessions, or 10x the historical max, is not a forecast - it is a bug. Clamp or reject it.
- Baseline fallback. If the model output fails a sanity check or the model errors, fall back to the seasonal-naive forecast from b1. A boring, correct baseline beats a spectacular wrong number every time.
- Fail loud, serve safe. Log the fallback so a human investigates - but never let the pipeline serve nonsense while it waits for one.
This is the same seasonal-naive line you built in b1, now earning its second job: the safety net under the whole system.
The closed loop back to b1 5 min live
Everything in this track now becomes a cycle. Forecast, observe the actuals, measure the error, ask whether it drifted, and either retrain or fall back to the baseline - then forecast again. A guardrail gate sits in the middle so no absurd number escapes. This is the loop that turns a one-off model into a living system, and it hands the series straight back to the analyst who first plotted it in b1.
LiveTriggering a retrain: schedule + event, together3 min▶
b9 gave you the scheduled clock. Monitoring adds the second, event-driven clock:
- Scheduled retrain (b9) - refit on cadence, regardless. The steady heartbeat.
- Event-driven retrain (b10) - fire off-schedule the moment rolling error crosses the threshold or the drift detector alerts. The reflex.
- Escalation - if a retrain does not restore accuracy, that is a concept-drift signal. Page the analyst - the model, features, or method may need to change, not just the data.
A rolling-error monitor that fires a retrain alert ★ 12 min · everyone builds
Build the error tracker: from a forecast-vs-actual log, compute rolling MAPE over the last N days and flag when it crosses a threshold. Crossing fires a "retrain" alert - the event-driven trigger from Part 2.
Simulate a forecast-vs-actual log: each day holds yesterday's forecast and the actual that arrived. Healthy at first, then the actuals drift up (the model starts missing).
Compute per-day absolute percentage error, then a rolling MAPE over the last N days.
Flag every day the rolling MAPE crosses the threshold, and raise a single retrain alert on the first crossing.
A simple drift detector on the actuals ★ 8 min · build your own
Error tells you something broke; drift on the inputs tells you the distribution moved. Build a lightweight detector that compares the recent actuals' distribution to the training distribution using the Population Stability Index (PSI) - a standard, dependency-free drift score - and raises an alert when it crosses a rule-of-thumb bound.
Take a training window (what the model learned on) and a recent window (the latest actuals).
Bin both on the training quantiles, then compute PSI across the bins.
Apply the usual rule of thumb: PSI < 0.1 stable, 0.1-0.25 moderate drift, > 0.25 significant - alert.
PSI is a data-drift alarm, not a diagnosis. A high PSI says the input distribution moved - it does not say whether the learned relationship still holds. Pair it with the error monitor from Demo 1: distribution shifted + error rose after a refit points to concept drift, and that is when you escalate to the analyst rather than just retraining. Keep it simple and honest; a dependency-free PSI you understand beats a black-box drift library you do not.
Try it yourself - close your own loop ◐ 40-60 min total
- Wire Demo 1 onto your b9 store: each day, append yesterday's forecast and the actual to a log, and compute rolling MAPE against your b8 backtested baseline.
- Set a threshold at ~1.3x your backtest MAPE and have a crossing fire a retrain alert (print, log, or a real notification).
- Add the PSI drift detector on your actuals and decide your own bound. When it and the error monitor both fire, tag it "investigate concept drift".
- Add a guardrail: sanity-bound the model output and fall back to seasonal-naive (from b1) when it is absurd or the model errors. Log the fallback.
- Draw your closed loop on one page - forecast → observe → measure → drift? → retrain / fall back - and name who gets paged at each gate.
- Capstone reflection: trace one forecast through all ten sessions, series to monitored loop, and write the one honesty rail you would defend hardest in a review.
Sources covered (cited)
Monitoring a live forecast is industry practice built on the evaluation canon this track taught. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · What is the difference between data drift and concept drift?
Data drift often yields to a refit on the new data; concept drift may need a new model or features. Both raise error, so monitor inputs too, not just the error.
2 · Why alert on rolling error rather than a single day's error?
Every series has off days. Alert on the rolling metric crossing a threshold, anchored to the backtest, so you catch real degradation without alert fatigue.
3 · What should a guardrail do when the model output is absurd?
A boring correct baseline beats a spectacular wrong number. Sanity-bound the output, fall back to seasonal-naive on failure, and page a human - never serve nonsense.