Builder session 5 · Isolation Forest - learn anomaly detection with phoebe

Part 0

A different question entirely

In b4 you scored points by how dense their neighbourhood was - kNN distance and the Local Outlier Factor. Those need a distance metric and slow down as the data grows. Tonight's method, Isolation Forest (Liu, Ting & Zhou, 2008), needs neither. It builds an ensemble of random trees that recursively cut the data with random splits, and simply counts how many cuts it takes to isolate each point. Anomalies fall out in a handful of cuts; normal points hide deep in the tree. It is fast, it scales to large and high-dimensional data, and it has become the default first reach for tabular anomaly detection.

Live - presented in session Self-study - read after class ★ Run it now (Python) Sources covered (cited)

★ What you walk out with today A feature matrix built from the Lumen series (value, lags, rolling stats), a fitted IsolationForest scored honestly with precision and recall, a feel for reading score_samples to rank the most anomalous days, and a hard-won understanding of what the contamination parameter really does - it sets your threshold.

Part 1 · covers Liu, Ting & Zhou 2008

Few and different isolate fast 6 min live

The core insight is almost too simple. Pick a random feature, pick a random split value, and cut. Repeat until every point sits alone. A point that is different from the rest gets separated in very few cuts; a point deep inside a dense cluster needs many. The number of cuts to isolate a point - its path length - is the anomaly signal. Short path means anomalous.

🔍 Click to zoom - short path isolates the outlier, long path buries the normal point

LiveWhy this beats distance methods at scale3 min▶

No distance metric - it only ever compares a value to a random split, so it sidesteps the curse of dimensionality that bloats kNN and LOF distances.
Near-linear time - each tree is built on a small subsample (256 points by default), so it stays fast on millions of rows and many features.
Ensemble, not one tree - a single random tree is luck; averaging path length over a forest of them gives a stable, smooth anomaly score.

Real world

Isolation Forest is the workhorse of tabular anomaly detection. Fraud teams, infrastructure monitoring, and manufacturing QA reach for it first because it needs almost no tuning, handles dozens of features, and does not assume any shape for normal. When someone says "we threw an anomaly model at it," this is usually the model.

Part 2 · from trees to a score

Average path length becomes the score 5 min live

One random tree is noisy - a lucky split could isolate a normal point early. So the forest builds many trees and averages each point's path length across all of them. Short average path across the ensemble means the point was consistently easy to cut off: anomalous. scikit-learn wraps this in score_samples (lower = more anomalous) and decision_function (the same, shifted so negative means flagged).

🔍 Click to zoom - the forest averages path length into a single anomaly score

Self-studyThe contamination trap2 min read▶

Isolation Forest produces a continuous score for every point. To turn that into a yes/no flag it needs a threshold - and that is exactly what contamination sets. Tell it contamination=0.05 and it will flag the lowest-scoring 5% of points, no matter what the data actually contains.

It is your assumed anomaly rate - a prior you supply, not something the model discovers.
Set it too high and you flood the business with false alarms; the detector gets muted (alert fatigue).
Set it too low and real anomalies slip through unflagged.
You rarely know the true rate - so treat contamination as a dial you tune against precision and recall, or skip the flag entirely and rank by raw score.

The safest habit: read the scores, rank the most anomalous days, and let a human set the cut where the cost of a miss balances the cost of a false alarm.

Demo 1 of 2

Fit Isolation Forest on Lumen ★ 12 min · everyone builds

Build a feature matrix from the Lumen series, fit the forest unsupervised, and score it against the ground-truth labels.

Rebuild the labeled Lumen series so the notebook runs standalone.

Engineer features: the value, a lag, and rolling mean and std - so the forest can see context, not just level.

Fit IsolationForest, read predict and score_samples, and score precision, recall, and F1.

★ Run it now (Python) - feature matrix + IsolationForestimport pandas as pd, numpy as np from sklearn.ensemble import IsolationForest # --- rebuild the labeled Lumen series (from b1) --- idx = pd.date_range("2026-01-01", periods=90, freq="D") season = np.array([30,42,48,40,22,-95,-110]); i = np.arange(90) y = 900 + 1.2*i + season[i % 7] + 10*np.sin(i*1.7) + 7*np.cos(i*0.6) truth = [20, 45, 63, 70, 71, 72] y[20] += 430; y[45] -= 380 y[63] = y[63] - season[63 % 7] + season[6] - 40 y[[70,71,72]] += [175,190,170] s = pd.Series(np.round(y), index=idx, name="sessions") # --- feature matrix: value + lags + rolling stats --- df = pd.DataFrame({"value": s}) df["lag1"] = s.shift(1) df["lag7"] = s.shift(7) df["rmean"] = s.rolling(7, min_periods=1).mean() df["rstd"] = s.rolling(7, min_periods=1).std() X = df.bfill().ffill().values # --- fit unsupervised on the features only --- iso = IsolationForest(n_estimators=200, contamination=0.07, random_state=0).fit(X) pred = iso.predict(X) # -1 = anomaly, 1 = normal score = iso.score_samples(X) # lower = more anomalous flags = set(np.where(pred == -1)[0]) # --- score vs truth (labels used ONLY for scoring) --- t = set(truth) tp = len(flags & t); fp = len(flags - t); fn = len(t - flags) p = tp/(tp+fp) if tp+fp else 0; r = tp/(tp+fn) if tp+fn else 0 f1 = 2*p*r/(p+r) if p+r else 0 print(f"caught {tp}/{len(t)} precision {p:.2f} recall {r:.2f} F1 {f1:.2f}")

Give the forest context features, not just the raw value A bare value column turns Isolation Forest back into a global threshold - it will catch the spike and drop but miss the contextual dip. Adding lags and rolling stats lets a random split separate "high for its recent neighbourhood," which is what a contextual anomaly is. Feature engineering is where most of the win lives.

Demo 2 of 2

Sweep contamination, then rank by score ★ 10 min · build your own

Contamination is the whole ballgame. Sweep a few values and watch precision and recall trade off, then drop the flag entirely and just rank the most anomalous days by their raw score - the honest, threshold-free view.

Fit the forest at several contamination values and score each.

Watch precision fall as contamination rises (more flags, more false alarms) and recall climb.

Rank days by score_samples to surface the most anomalous - no threshold needed.

★ Run it now (Python) - the contamination sweep + rankingimport numpy as np, pandas as pd from sklearn.ensemble import IsolationForest def score(flags, truth): f, t = set(flags), set(truth) tp = len(f & t); fp = len(f - t); fn = len(t - f) p = tp/(tp+fp) if tp+fp else 0 r = tp/(tp+fn) if tp+fn else 0 return p, r, (2*p*r/(p+r) if p+r else 0) # sweep the assumed anomaly rate - it sets the threshold, nothing else for c in [0.03, 0.05, 0.07, 0.12, 0.20]: iso = IsolationForest(n_estimators=200, contamination=c, random_state=0).fit(X) fl = np.where(iso.predict(X) == -1)[0] p, r, f1 = score(fl, truth) print(f"contamination {c:.2f} flagged {len(fl):2d} P {p:.2f} R {r:.2f} F1 {f1:.2f}") # threshold-free: rank the most anomalous days by raw score (lower = worse) iso = IsolationForest(n_estimators=200, contamination="auto", random_state=0).fit(X) ranked = pd.Series(iso.score_samples(X), index=s.index).sort_values() print("\nmost anomalous days:") print(ranked.head(6))

Real world

Ranking beats flagging when you own the response. A fraud analyst does not need a binary flag - they need a queue sorted worst-first, and they work down it until their time runs out. Handing the raw score to a human sidesteps the contamination guess entirely, and it is why most mature detection systems surface a ranked list, not a red/green light. Reserve hard flags for fully automated actions where a person cannot be in the loop.

Homework

Try it yourself - this week ◐ 30-45 min total

Fit Isolation Forest on your own feature matrix and compare its F1 to the z-score baseline from b1. Does it earn its complexity?
Drop the lag and rolling features and refit on the raw value alone. Watch it lose the contextual anomaly, and explain why.
Sweep contamination from 0.02 to 0.25 and plot precision vs recall. Mark the value that matches your real cost of a miss vs a false alarm.
Rank the days by score_samples and check whether the top of the list matches the true anomalies - the threshold-free view.
Optional: read Liu, Ting & Zhou (2008), "Isolation Forest" - the path-length derivation is short and elegant.

Source material

Sources covered (cited)

Taught from the detection canon. This session covers:

✓Liu, Ting & Zhou (2008), "Isolation Forest," IEEE ICDM 2008Parts 1-2 · isolation by random splits, path length as anomaly score

◐scikit-learn "Novelty and Outlier Detection" user guide - IsolationForestDemos 1-2 · fit, score_samples, and the contamination parameter

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · What does Isolation Forest use as its anomaly signal?

Anomalies are few and different, so random splits cut them off in few steps. The average path length across the forest is the score: short = anomalous.

2 · What does the contamination parameter actually control?

Contamination is your assumed anomaly rate. It just sets where the score is cut into flag/no-flag. Too high floods false alarms; too low misses real anomalies.

3 · Why give Isolation Forest lag and rolling features instead of the raw value alone?

On the raw value alone, the forest is just a global threshold and misses the contextual dip. Context features let a random split isolate points that are odd for their moment.

Builder session 5 cheat sheet · pin this

Core ideaanomalies are few and different, so random splits isolate them in fewer steps.

Path lengththe number of cuts to isolate a point. Short path = anomalous.

Ensembleaverage path length over many random trees; one tree is luck, the forest is stable.

Fast & scalableno distance metric, near-linear time, handles large and high-dimensional data.

score_samplescontinuous anomaly score, lower = more anomalous. Rank days with it.

contaminationyour assumed anomaly rate - it just sets the flag threshold, so tune it honestly.

Feature it wellvalue + lags + rolling stats, or it collapses to a global threshold.

Next sessionb6 - one-class & covariance: learning the shape of normal only.