Builder session 9 · Evaluation - learn anomaly detection with phoebe

Part 0

Why evaluation is the hardest part

You have eight sessions of detectors - statistical, rolling, distance, isolation, one-class, time-series, deep. Tonight is the referee. On rare events, the metric everyone reaches for first is worse than useless: it actively rewards a detector for doing nothing. So we throw accuracy out and build the scoreboard that survives a 1% base rate - precision, recall, F1, and the precision-recall curve summarised as PR-AUC. Then we pick the operating threshold on purpose, from the business cost of a miss versus a false alarm, and handle the one time-series wrinkle that trips people up: how to score a detection that lands inside, but not exactly on, a true anomalous segment. This is the same math the b3 live detector shows moving as you turn the sensitivity dial.

Live - presented in session Self-study - read after class ★ Run it now (Python) Sources covered (cited)

★ What you walk out with today Precision, recall, and F1 computed from scratch and understood; a full precision-recall curve and PR-AUC from a continuous anomaly score, and the reason PR beats ROC under heavy imbalance; a defensible way to choose the operating threshold from real costs; and a working point-adjustment scorer for time-series segments, with clear eyes on how it can inflate the numbers.

Part 1 · precision, recall, F1

Accuracy is out - these three are in 6 min live

Recall the base-rate trap from b1: with anomalies under 1%, a detector that flags nothing scores 99%+ accuracy and catches zero. So accuracy never gets a vote. The honest three come from the confusion matrix on the positive (anomaly) class alone: precision asks, of the days you flagged, how many were real; recall asks, of the real anomalies, how many you caught; F1 is their harmonic mean, the single number that punishes a detector for being lopsided.

LiveThe four cells, and why F1 not accuracy3 min▶

Count four things against the labels, and notice what each metric ignores:

TP flagged and real · FP flagged but normal (the false alarm) · FN real but missed (the costly one) · TN normal and left alone.
Precision = TP / (TP + FP) - the false-alarm side. Low precision is what causes alert fatigue: the team stops trusting the pager.
Recall = TP / (TP + FN) - the miss side. Low recall is the fraud that slipped through, the outage nobody paged on.
F1 = 2PR / (P + R) - the harmonic mean, so a detector cannot win by acing one and tanking the other. Accuracy leans on TN, and TN is enormous and boring here, which is exactly why it lies.

Real world

Extreme imbalance is the normal case, not an edge case. Card fraud runs near 0.1%, hardware failures far rarer. With 999 normal to 1 anomaly, one lucky true catch amid a flood of false alarms still looks fine to accuracy. Precision and recall report the flood; F1 refuses to hide it. Always report the base rate next to the score, so a reader can judge whether the number is impressive or trivial.

Part 2 · the curve and its area

The precision-recall curve and PR-AUC 5 min live

Most detectors do not output a yes or no - they output a score (isolation-forest anomaly score, seasonal residual size, LOF value). Every threshold you could set on that score gives one precision and one recall. Sweep the threshold from strict to loose and you trace the precision-recall curve: high precision and low recall on the left, sliding to high recall and low precision on the right. Its area, the PR-AUC (average precision), is the single number that summarises the detector across all thresholds - and on heavy imbalance it is the honest one, not ROC-AUC.

🔍 Click to zoom - the precision-recall curve, its area, and the chosen operating point

Self-studyWhy PR-AUC beats ROC-AUC on heavy imbalance2 min read▶

Both curves sweep a threshold, but they measure different things:

ROC plots recall against the false-positive rate, FP / (FP + TN). With a giant normal class, TN is enormous, so even thousands of false alarms barely move the false-positive rate. ROC-AUC stays flatteringly high while the pager screams.
PR plots precision against recall. Precision has FP in its denominator without the TN cushion, so every false alarm shows up. The curve, and PR-AUC, react to exactly the failure you care about.
The random-chance floor also differs: for ROC it is a fixed diagonal; for PR it is the base rate, so a PR-AUC of 0.4 on a 1% base rate is a genuinely strong detector, not a weak one. Judge PR-AUC relative to the base rate.

Demo 1 of 2

Precision, recall, F1, then the whole PR curve ★ 12 min · everyone builds

Compute the three metrics by hand for one set of flags, then feed a continuous anomaly score to sklearn and get the entire precision-recall curve and PR-AUC.

Rebuild the labeled Lumen series so the notebook runs standalone.

From a fixed detector's flags, count TP/FP/FN and compute precision, recall, F1 from scratch.

Take an IsolationForest anomaly score and sweep every threshold with precision_recall_curve.

Summarise it with average_precision_score (PR-AUC) and compare it to the base rate.

★ Run it now (Python) - metrics from scratch + PR-AUCimport pandas as pd, numpy as np from sklearn.ensemble import IsolationForest from sklearn.metrics import precision_recall_curve, average_precision_score # --- rebuild the labeled Lumen series (standalone) --- idx = pd.date_range("2026-01-01", periods=90, freq="D") season = np.array([30,42,48,40,22,-95,-110]) i = np.arange(90) y = 900 + 1.2*i + season[i % 7] + 10*np.sin(i*1.7) + 7*np.cos(i*0.6) y = y.astype(float) truth = [20, 45, 63, 70, 71, 72] y[20] += 430; y[45] -= 380 y[63] = y[63] - season[63 % 7] + season[6] - 40 y[[70,71,72]] += [175,190,170] s = pd.Series(np.round(y), index=idx, name="sessions") label = np.zeros(90, int); label[truth] = 1 # --- precision / recall / F1 from scratch, for one fixed set of flags --- z = (s - s.mean()) / s.std() flags = (np.abs(z) > 3).astype(int) tp = int((flags & label).sum()) fp = int((flags & (1 - label)).sum()) fn = int(((1 - flags) & label).sum()) precision = tp / (tp + fp) if tp + fp else 0.0 recall = tp / (tp + fn) if tp + fn else 0.0 f1 = 2*precision*recall / (precision + recall) if precision + recall else 0.0 print(f"z>3 TP={tp} FP={fp} FN={fn} P={precision:.2f} R={recall:.2f} F1={f1:.2f}") # --- a continuous anomaly SCORE -> the whole PR curve --- X = s.values.reshape(-1, 1) iso = IsolationForest(contamination=0.06, random_state=0).fit(X) score = -iso.score_samples(X) # higher = more anomalous prec, rec, thr = precision_recall_curve(label, score) pr_auc = average_precision_score(label, score) print(f"PR-AUC (average precision) = {pr_auc:.3f} base rate = {label.mean():.3f}")

Report PR-AUC next to the base rate, always A PR-AUC means nothing on its own. On a 6.7% base rate it must clear 0.067 just to beat random. Print them together so nobody reads a real result as a bad one, or a trivial one as a triumph. This is the same trap accuracy sets, one level up.

Part 3 · the time-series wrinkle

Point-adjustment: scoring a segment, not a point 5 min live

Time-series anomalies come in segments - an outage lasts an hour, a bot surge runs for a run of days. If your detector flags the middle of a true anomalous segment but not its exact first point, point-wise scoring calls the rest misses, which feels unfair: you did catch the incident. The common convention, point-adjustment, fixes this: if any point inside a true segment is flagged, the whole segment counts as detected. It is standard in time-series anomaly papers - and it can quietly inflate your scores, so use it with your eyes open.

🔍 Click to zoom - point-adjustment turns one hit inside a true segment into a full catch

Self-studyUse it, but guard against the inflation2 min read▶

Point-adjustment is the near-universal convention in time-series anomaly benchmarks, and it does encode a real truth: catching an incident anywhere inside its window usually means the on-call team is paged in time. But the failure mode is sharp:

The longer the segment, the bigger the free lunch. One flagged point turns a 100-point outage into a perfect catch. A detector that fires almost at random can score deceptively well.
Guardrails - report point-wise scores alongside; require the hit to fall within a delay budget (paged fast enough to matter); or cap the credit per segment. Never publish point-adjusted numbers on their own.
When you compare detectors, hold the scoring convention fixed. A model that looks better only under point-adjustment may just be luckier at landing one point in long segments.

Demo 2 of 2

Implement point-adjustment and watch it inflate ★ 8 min · build your own

Expand the labels into segments, mark a segment caught if any point inside it was flagged, and score both ways side by side. The gap you see is the inflation you must always disclose.

Write segments to find contiguous runs of anomaly labels.

Write point_adjust to flag a whole true segment when any point in it is flagged.

Score point-wise and point-adjusted with the same function and compare recall and F1.

★ Run it now (Python) - point-adjustment vs point-wiseimport numpy as np # ground truth: point anomalies on days 20, 45, 63 and a collective segment 70-72 label = np.zeros(90, int); label[[20, 45, 63, 70, 71, 72]] = 1 # a detector that caught the spike, the drop, and only ONE day of the run flags = np.zeros(90, int); flags[[20, 45, 71]] = 1 def segments(label): """Inclusive (start, end) ranges of contiguous 1s in the label array.""" segs, start = [], None for j, v in enumerate(label): if v and start is None: start = j elif not v and start is not None: segs.append((start, j - 1)); start = None if start is not None: segs.append((start, len(label) - 1)) return segs def point_adjust(flags, label): """If any point inside a true segment is flagged, flag the whole segment.""" adj = flags.copy() for a, b in segments(label): if flags[a:b + 1].any(): adj[a:b + 1] = 1 return adj def score(flags, label): tp = int((flags & label).sum()); fp = int((flags & (1 - label)).sum()) fn = int(((1 - flags) & label).sum()) p = tp / (tp + fp) if tp + fp else 0.0 r = tp / (tp + fn) if tp + fn else 0.0 return p, r, (2*p*r / (p + r) if p + r else 0.0) p0, r0, f0 = score(flags, label) # point-wise p1, r1, f1 = score(point_adjust(flags, label), label) # point-adjusted print(f"point-wise P={p0:.2f} R={r0:.2f} F1={f0:.2f}") print(f"point-adjust P={p1:.2f} R={r1:.2f} F1={f1:.2f}") # recall jumps from 0.50 to 0.83 on one extra flagged point - report both, never just the second

Real world

This is the meter behind the b3 live detector. When you slide its sensitivity and watch precision and recall trade off, you are walking the precision-recall curve by hand, one threshold at a time. Everything tonight is what those two live meters mean. And the honest habit is the same at every scale: quote precision, recall, F1, PR-AUC, the base rate, and your scoring convention together - a number without its context is how detectors get oversold.

Homework

Try it yourself - this week ◐ 30-45 min total

Compute precision, recall, and F1 from scratch for two of your detectors, and confirm they match sklearn's precision_recall_fscore_support.
Build the full precision-recall curve from an IsolationForest or seasonal-residual score and read off the threshold that gives you 90% recall - note the precision you pay for it.
Compute both PR-AUC and ROC-AUC on the same score and see how much rosier ROC looks. Explain why to a teammate.
Run the point-adjustment scorer and quantify the inflation on the collective segment. Add a delay-budget guardrail and see the score come back down.
Optional: skim the NAB scoring methodology and note how it rewards early detection inside a window - a cousin of point-adjustment.

Source material

Sources covered (cited)

Taught from the anomaly-detection canon. This session covers:

✓Chandola, Banerjee & Kumar (2009), "Anomaly Detection: A Survey," ACM Computing SurveysParts 1-3 · evaluation on rare events, precision/recall over accuracy

◐Numenta Anomaly Benchmark (NAB, Lavin & Ahmad 2015)Part 3 · scoring windows for streaming time-series anomalies

◐scikit-learn metrics user guide (precision_recall_curve, average_precision_score)Demo 1 · the PR curve and PR-AUC in practice

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · Why is PR-AUC preferred over ROC-AUC on heavily imbalanced anomaly data?

With a giant TN, ROC-AUC stays high even amid many false alarms. Precision has no TN cushion, so the PR curve and PR-AUC react to the failure you actually care about.

2 · How should you choose the operating threshold on the PR curve?

The threshold is a business decision, not a default. If a miss is far costlier than a false alarm, move toward recall; if the team is drowning in false alarms, move toward precision.

3 · What is the main caveat of point-adjustment?

The longer the segment, the bigger the free lunch - one hit can score a 100-point outage as a perfect catch. Report point-adjusted next to point-wise, never alone.

Builder session 9 cheat sheet · pin this

Accuracy is outon rare events it rewards "flag nothing". Never report it alone.

PrecisionTP / (TP + FP). Of your flags, how many were real. Low precision → alert fatigue.

RecallTP / (TP + FN). Of the real anomalies, how many you caught. Low recall → misses.

F12PR / (P + R). The harmonic mean; punishes a lopsided detector.

PR curvesweep the score threshold; each cut is one (recall, precision) point.

PR-AUCarea under it (average precision). Judge it against the base rate, not against 1.0.

PR > ROC hereROC's FP rate is diluted by the huge normal class; PR isn't.

Thresholdpick it from cost of a miss vs a false alarm, not a default.

Point-adjustmentany hit inside a true segment counts the whole segment - report it next to point-wise.

Next sessionb10 - productionizing & alerting: streaming, dedup, feedback, drift.