The one number never to trust
Every detector your org buys or builds will come with a score attached, and the person presenting it wants you to be impressed. Your job as a leader is not to read the model code - it is to ask for the right numbers and to smell it when the wrong ones are being waved around. This session is that filter. It is short, it has no math, and it will change what you ask in every detection review from now on. The headline: on rare events, accuracy is a liar, and the honest story is told by two numbers pulling against each other.
Why accuracy lies 4 min live
Anomalies are rare by definition - fraud, outages, and defects are usually well under 1% of all the events you see. That rarity breaks the metric everyone reaches for first. A detector that flags nothing at all is right about 99%+ of the time, because 99%+ of days really are normal. It scores beautifully on accuracy and catches zero real problems. This is the base-rate trap, and it is why "99% accurate" should make you suspicious, not comfortable.
LiveThe question to ask in the room2 min▶
When a team presents a detector, the sentence "it is 95% accurate" tells you almost nothing on a rare-event problem. Two questions cut through it:
- "Of everything it flagged, how much was really a problem?" - that is precision, the false-alarm question.
- "Of the real problems that happened, how many did it catch?" - that is recall, the miss question.
If they cannot answer both, they do not yet know whether the detector works. Accuracy alone can hide a detector that flags nothing, or one that flags everything.
Fraud, outages, and disease screening all live here. A rare-disease test that always says "healthy" is over 99% accurate and lethal. A fraud model that says "no fraud" on every transaction is 99.9% accurate and bankrupts you. The rarer the thing you are hunting, the more accuracy flatters a detector for looking the other way.
Precision, recall, and their balance 7 min live
Two numbers replace the lying one, and they answer the two things you actually care about. Precision is about false alarms: of everything the detector flagged, what share was truly anomalous. Recall is about misses: of everything that was truly anomalous, what share the detector caught. F1 is a single number that balances the two, so a detector cannot win by acing one and failing the other.
LivePrecision - the false-alarm rate2 min▶
Precision answers: when this thing cries wolf, how often is there actually a wolf? Low precision means most alerts are noise - the team opens ten tickets and nine are nothing. That is expensive in attention, and, as we will see, it is how a detector gets muted.
A payments team lives on precision. Every false fraud flag freezes a real customer's card and generates an angry call. If only one flag in five is real fraud, the cost of the four false alarms - support time, churned customers - can dwarf the fraud they caught. Precision is the number their leadership watches.
LiveRecall - the miss rate2 min▶
Recall answers: of the real problems that happened, how many did we catch? Low recall means anomalies slip through silently. You do not get a ticket, you get a post-mortem. Recall is what you care about most when a miss is catastrophic.
An outage-detection team lives on recall. A missed outage is a customer-facing incident and a headline. They will happily tolerate more false alarms to make sure a real outage is never missed - so they tune for high recall, and accept the precision cost. The right balance depends entirely on which mistake hurts more.
Self-studyWhy F1, and why not accuracy2 min read▶
Precision and recall can be gamed one at a time. Flag everything and recall hits 100% while precision collapses. Flag almost nothing and precision looks high while recall collapses. F1 combines them into one score that only rises when both are healthy, so it is the honest single number to compare two detectors.
You may also hear PR-AUC (the area under the precision-recall curve) - a summary of how the detector performs across every possible sensitivity setting, not just one. For a leader, the takeaway is simpler: ask for precision, recall, and F1 together. If someone offers you accuracy on a rare-event problem, that is the tell.
The precision/recall trade-off 6 min live
Here is the part most people miss: precision and recall are not fixed properties of a detector. They sit on a dial - the sensitivity threshold. Crank the detector to fire more easily and you catch more real anomalies (recall up) but drown in false alarms (precision down). Turn it down and the alerts get cleaner (precision up) but real anomalies slip through (recall down). Where you set that dial is a business decision, not a model detail - and it is yours.
LiveSetting the dial is your decision, not the model's3 min▶
The model produces a score for how odd each event is. Turning that score into a yes/no alert needs a threshold - and that threshold trades precision against recall. There is no "correct" setting a model can find for you, because the right setting depends on which mistake costs your business more:
- When a miss is catastrophic (an outage, a safety defect), set the dial sensitive - accept more false alarms to protect recall.
- When false alarms are expensive (freezing real customers, waking an on-call engineer), set it conservative - protect precision, accept some misses.
Ask your team to show you precision and recall at a few different threshold settings, then you pick the operating point. That is the leader's lever, and it is the subject of the next session.
Alert fatigue kills detectors 3 min live
Here is the failure mode that no model metric shows you. A detector with low precision cries wolf constantly. The humans on the other end learn that most alerts are nothing, so they start ignoring them - muting the channel, auto-closing the tickets, skimming past the red. And then the one real anomaly arrives, fires exactly like all the false ones, and gets ignored with the rest. Low precision does not just waste time. It destroys trust in the whole system, and a muted detector catches nothing.
Self-studySymptoms of a detector nobody trusts anymore2 min read▶
Alert fatigue rarely announces itself. It shows up as small, rational human behaviours:
- A Slack channel or inbox rule that quietly filters the detector's alerts out of sight.
- Tickets auto-closed or bulk-dismissed without investigation.
- "Oh, that one always fires, ignore it" said out loud in a standup.
- Nobody can remember the last time an alert led to a real fix.
Any of these means the detector's effective recall is already near zero, whatever the dashboard claims. The fix is not a better model - it is raising precision (turn the dial down) until each alert is worth a human's attention again.
Try it yourself - this week ◐ 20-30 min total
- Pick one detector or alerting system your org actually relies on (fraud, outage, quality, security, or a business KPI alert).
- Ask for its precision: of everything it flagged last month, what share was truly a problem? If nobody knows, that is your finding.
- Ask for its recall: of the real problems that happened last month, how many did it actually catch? Again, "we don't measure that" is a finding.
- Now the human question: is anyone muting it? Look for filtered channels, auto-closed tickets, or "that one always fires." If people are ignoring it, its true recall is near zero.
- Write one sentence: is this detector's problem too many false alarms (precision), too many misses (recall), or lost trust (fatigue) - and which way should the dial move?
- Optional: skim the framing in the Numenta Anomaly Benchmark on why streaming detectors must be judged by more than accuracy.
Sources covered (cited)
Taught from the anomaly-detection canon, framed for decision-makers. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · A vendor says their detector is "99% accurate" on a problem where anomalies are 1% of events. Why is that unimpressive?
The base-rate trap. When anomalies are rare, doing nothing scores near-perfect accuracy. Ask for precision and recall instead.
2 · Your team cranks the detector's sensitivity up to catch more real anomalies. What predictably happens?
Sensitivity is a dial. More recall costs precision. Where you set it is a business decision about which mistake hurts more.
3 · In practice, what most often kills a detection system that looks fine on paper?
Low precision spends the team's trust. Once alerts are muted, the detector's real recall collapses no matter what the offline score says.