The one idea for today
The most common way an anomaly-detection project fails is not a bad model - it is a broken relay. The data scientist ships a detector that never gets served; the alerts fire into a channel no one owns; on-call mutes them and no one hears; the confirmations of "real" or "false" evaporate instead of improving the detector. Detection is a five-role relay, and the value lives in the hand-offs between roles. This session names the roles, the artifact each passes on, and the one loop - unique to detection - that turns responses back into a better detector.
Five roles, five hand-offs 12 min live
Follow the baton. Each role does its part and hands a specific artifact to the next - labels and a baseline, then a tuned detector, then live alerts, then triaged incidents, then a health view. Miss any hand-off and the relay stalls. The numbers in brackets map each role to the builder-track sessions where it lives.
LiveThe relay, role by role4 min▶
- Data analyst - profiles the history, labels the incidents everyone remembers, and ships a plain baseline detector (a z-score or seasonal rule). Hands off: labels + a baseline. This is the honest bar everything later must beat. [b1, b2]
- Data scientist - builds and tunes the real detector - statistical, ML, or deep - and, critically, sets the threshold on a precision/recall trade-off. That threshold is a business decision dressed as a technical one. Hands off: a tuned detector and a threshold. [b3-b9]
- ML engineer / DevOps - takes the detector off the notebook and streams it: runs detection on live data and routes each alert to a channel someone owns. Hands off: streaming alerts. [b10]
- On-call / ops - the humans who receive the alerts, respond, and - the part teams forget - record whether each one was real or a false alarm. Hands off: triaged incidents + feedback. [b10]
- Everyone, led by the data scientist - monitors the detector itself: is it still catching real anomalies, and is it crying wolf? Hands off: a health dashboard. [b10]
Self-studyWhere relays snap2 min read▶
Each hand-off is a place the whole system can quietly break:
- Analyst → DS - no labels, so no one can measure whether the detector is any good. The DS flies blind.
- DS → engineering - a great notebook detector that is never served. It scores well in a slide and catches nothing in production.
- Engineering → on-call - alerts fire into an unowned channel. Everyone assumes someone else is watching.
- On-call → back to DS - the disposition of each alert is never captured, so the detector never learns. This broken loop is the most common and most damaging gap.
The RACI grid 7 min live
The relay tells you the order; RACI tells you who owns each step. R does the work, A is accountable for the outcome, C is consulted. Notice the diagonal: accountability moves along the relay, and the data scientist stays consulted almost everywhere - because the threshold and the detector's health are theirs to defend.
| Step | Analyst | Data Scientist | ML Eng / DevOps | On-call / Ops |
|---|---|---|---|---|
| Label history + ship baseline | R / A | C | - | - |
| Build + tune detector, set threshold | C | R / A | C | - |
| Stream detection + route alerts | - | C | R / A | C |
| Respond to alerts (disposition) | - | C | C | R / A |
| Monitor the detector | C | A | C | R |
Self-studySmall team? One person wears several hats2 min read▶
The grid names five roles, not five people. On a small team one analyst-turned-data-scientist may cover the first two columns, and a platform engineer may own serving and monitoring both. That is fine - what matters is that every step has a clear Accountable, not that every role has a distinct headcount. Two rules survive any team size:
- No orphan steps. Walk the five rows and check each has exactly one A. The row that usually goes orphaned is "monitor the detector" - everyone assumes the model still works until it quietly stops.
- Never merge the loop away. Even when one person plays both on-call and data scientist, the disposition still has to be written down, not just remembered. The feedback loop is a data flow, not a conversation - if it lives only in someone's head, it is already lost.
The point of RACI here is not bureaucracy; it is to surface the two gaps that sink detection projects - an unowned monitoring step and an uncaptured feedback loop - before they cost you a missed outage.
On-call feedback retrains the detector 6 min live
Here is what makes an anomaly pipeline different from an ordinary dashboard: the last role feeds the first. Every time on-call marks an alert "real" or "false alarm," they are creating exactly the labelled data the whole system was starving for. Captured, that disposition becomes precious training signal - the false alarms teach the detector to fire less, the confirmed hits teach it what a real one looks like. Left uncaptured, the detector never improves and slowly drifts into noise.
Lumen's on-call loop. An alert fires: sessions dropped hard at 2am. On-call checks, finds a CDN outage - real - and marks it. Next week a similar-looking dip fires; on-call finds a scheduled maintenance window - false alarm - and marks that too. Those two dispositions flow back to the data scientist, who now has labelled examples to retune the threshold and add a maintenance-window rule. Without that loop, the same false alarm fires every month until on-call mutes the whole channel - and then the next real outage goes unseen. The feedback loop is not a nice-to-have; it is what keeps the detector alive.
Self-studyWhy this loop is unique to detection2 min read▶
In most analytics, the humans consume the output and that is the end of the line. In detection, the humans are the label source. Because true anomalies are rare and their truth only emerges through investigation, the on-call disposition is often the only ground truth the system will ever get. That is why the loop matters so much more here than in a normal reporting pipeline - and why the single most valuable thing a leader can do is make sure alert dispositions are captured, not lost. If your pipeline has no path from on-call back to the detector, you do not have a detection system; you have an alarm no one can improve.
Try it yourself - this week ◐ 20-30 min total
- Take one detection need in your org and map real names onto the five roles: analyst, data scientist, ML engineer/DevOps, on-call/ops, monitoring.
- For each hand-off, name the artifact that actually gets passed. Where a box is empty, you have found a gap.
- Trace the feedback loop specifically: when an alert fires and someone decides "real" or "false," where does that decision go? If the answer is "nowhere," that is your highest-value fix.
- Ask who owns the threshold. If no single person is accountable for how often it pages, that is a second gap.
- Optional: sketch the RACI grid for your own team and circle the row with no clear A.
Sources covered (cited)
This session is drawn from applied detection practice, with the evaluation framing that anchors the roles:
Three questions before you go 🎯 ◐ 90 seconds
1 · The data scientist hands the ML engineer a tuned detector. What did the analyst hand the data scientist first?
The analyst profiles history, labels the incidents, and ships a baseline - the honest bar the data scientist's detector must beat.
2 · What makes the anomaly-detection pipeline different from an ordinary reporting dashboard?
Detection has a feedback loop: on-call marking alerts real or false creates the scarce labelled data that improves the detector. That loop is the defining feature.
3 · Who is accountable for the alert threshold, and why is it a business decision?
The threshold is a precision/recall trade-off. The data scientist owns it but sets it with the people who live with the alerts, which is why they are consulted across the grid.