From notebook to on-call pager
Nine sessions built and measured detectors on a fixed series sitting still in memory. Production is different in every way that matters: points arrive one at a time, latency counts, the baseline drifts, and a false alarm is not a number on a chart but a human woken at 3am. Tonight we ship. We make the detector streaming so it scores each new point online against a rolling baseline; we build the alerting layer that turns raw flags into one trustworthy page per incident; we wire the feedback loop so the people who answer those pages teach the threshold what a real anomaly looks like; and we monitor the detector itself for drift, because "normal" never stops moving. The single enemy the whole time is alert fatigue - the moment the team mutes you, your recall is effectively zero no matter what the offline score said.
Score online, alert once, close the loop 6 min live
A production detector is a loop, not a script. Each new point is scored against a rolling or seasonal baseline that travels with the data; if it crosses the band, the flag enters an alerting layer that deduplicates a burst into one incident, tags severity, and routes it to the right on-call. The responder's verdict - real or false alarm - flows back as a label that tunes the threshold and feeds the next retrain. And before any automated high-stakes action, a guardrail gate hands off to a human.
LiveThe four jobs of the alerting layer3 min▶
Raw flags are not alerts. Between the detector and the pager sits a layer that does four things, and skipping any one of them is how a good detector still gets muted:
- Deduplicate - a three-day bot surge is one incident, not three pages. A cooldown or suppression window collapses a burst into a single alert with a running count.
- Prioritise - severity from how far past the band, and from what the metric feeds. A checkout-flow drop is a page; a blog-traffic wobble is a ticket.
- Route - the right on-call for the right system, with escalation if unacknowledged. An alert nobody owns is an alert nobody actions.
- Suppress - during known maintenance or a declared incident, mute to avoid a storm of redundant pages.
Alert fatigue kills more detectors than bad math. A detector that pages ten times a night for nine false alarms gets a mute rule within a week, and then it catches nothing - your real recall is zero regardless of the offline F1. Deduplication and a threshold tuned toward precision are not polish; they are what keeps the system alive.
A streaming detector with deduplication ★ 12 min · everyone builds
Replay the Lumen series as a live stream, one point at a time. Maintain a rolling baseline, alert when a point leaves the band, and suppress repeat alerts inside a cooldown so one incident is one page.
Rebuild the Lumen series and replay it point by point, as it would arrive in production.
Keep a fixed-size rolling window; score each new point before adding it to the baseline.
Emit an alert with a severity when a point crosses the band - but only if the cooldown has elapsed.
Confirm the collective run fires one deduplicated alert, not three.
Let the pager teach the threshold 5 min live
A shipped detector is not done - "normal" drifts as the product grows, seasons turn, and traffic mix shifts, so a threshold that was precise in spring floods the pager by autumn. The fix is a feedback loop: every alert comes back marked real or false by the responder, and those precious labels become a rolling precision you watch like a vital sign. When it slides below a floor, you have caught alert fatigue before the team mutes you - time to re-tune the threshold or retrain.
Self-studyConcept drift, retrain cadence, and guardrails2 min read▶
Three habits keep a live detector trustworthy long after launch:
- Monitor the detector, not just the metric. Track rolling precision (and recall, as investigations resolve) on the dispositions coming back. A quietly falling precision is concept drift showing up before anyone complains.
- Retrain on a cadence and on a trigger. Refit the baseline on a schedule (rolling window, seasonal refit) and also on demand when the drift monitor fires. Version the threshold so you can roll back a bad tune.
- Guardrail the high-stakes path. A detector can open a ticket or throttle a bot on its own; it should not lock a customer's account or halt a payment line without a human. Keep a person in the loop wherever a false alarm is expensive or irreversible.
Alert feedback + a drift early-warning ★ 8 min · build your own
Accumulate on-call dispositions into a rolling precision, and raise a retrain flag the moment it drops below a floor. This is the alert-fatigue smoke detector - it fires before the team reaches for the mute button.
Feed in the stream of dispositions (real or false alarm) as they come back from the pager.
Keep a rolling window of the last N verdicts and compute precision over it.
Raise a "retrain" flag when the rolling precision falls below the floor.
This is the loop the analyst from session 1 lives inside. They set the baseline and label the incidents; the detector scores the stream; the alerting layer pages once; the responder's verdict flows back as a fresh label; the drift monitor decides when to retrain - and it lands right back at the analyst. Ten sessions ago an anomaly was "a point that does not fit the pattern." Now it is a governed, measured, self-correcting system that a real on-call team can trust at 3am. That is the whole job.
Try it yourself - this week ◐ 30-45 min total
- Run the streaming detector on the Lumen series and confirm the collective run fires exactly one deduplicated alert. Then shorten the cooldown and watch the duplicates return.
- Swap the rolling baseline for an EWMA or a seasonal-residual baseline (from b3) inside the streaming loop and compare which anomalies each catches online.
- Feed a stream of dispositions where false alarms slowly increase, and tune the drift monitor's window and floor so it fires early but not on noise.
- Add severity-based routing: high severity pages, warn opens a ticket. Add a suppression window for a simulated maintenance period.
- Optional: read the NAB scoring methodology and note how it rewards early detection inside an anomaly window - the production cousin of low-latency streaming.
Sources covered (cited)
Taught from established production practice and the streaming-detection literature. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · In an online streaming detector, when should the rolling baseline be updated?
Score first, then append. If the anomaly enters the baseline before it is measured, it inflates the band and hides itself during exactly the events you built the detector to catch.
2 · Why is deduplication essential in the alerting layer?
One incident, one alert. Without a cooldown or suppression window, a three-day surge pages three times, and the fastest way to get muted is to cry wolf.
3 · What does a falling rolling precision on returned dispositions tell you?
Rising false alarms mean the world drifted out from under the threshold. Catching it in the rolling precision lets you act before alert fatigue sets in and the detector gets muted.