Builder session 4 · Distance & density - learn anomaly detection with phoebe

Part 0

A different question

So far every detector has drawn a global line - a z-score band, a Tukey fence, a seasonal residual cutoff. Distance and density methods throw that away and reason locally: an anomaly is a point with no close company. Two ideas carry the session. First, kNN distance - score each point by how far its nearest neighbours are; anomalies sit in empty space. Second, the Local Outlier Factor (Breunig et al., 2000) - a point is anomalous if its local density is much lower than its neighbours' density, which lets it find outliers even when different regions of the data are packed at different tightness. Both are naturally multivariate: they shine when you have several features, so for a time series we build features first - value, a lag, rolling statistics - and let the geometry do the work.

Live - presented in session Self-study - read after class ★ Run it now (Python) Sources covered (cited)

★ What you walk out with today A feature matrix built from the Lumen series, a kNN-distance detector scored against the truth, a LOF detector on the same features, and a clear, coded demonstration of the one thing LOF does that a global distance cannot: keep a low-density-but-normal point from being wrongly flagged.

Part 1 · distance to the k-th neighbour

How lonely is this point? 7 min live

The simplest neighbour-based score: measure the distance from each point to its k-th nearest neighbour, or the mean distance to its k nearest. Points deep inside a cluster have tiny distances; a point out in empty space has a large one. Sort by that score and the anomalies rise to the top.

🔍 Click to zoom - kNN distance: anomalies sit in empty space

LiveMultivariate by nature - and the dimensionality caveat3 min▶

Neighbour methods measure distance in feature space, so they earn their keep when a point is only anomalous across several features at once - a normal traffic volume at a normal hour but from an abnormal device mix. On a raw one-dimensional series they collapse back to something z-score-like, which is why we engineer features first:

value - the raw session count for the day.
lag - yesterday's value, so a sudden jump becomes a distance.
rolling mean / std - the recent local level and volatility, so a point far from its own neighbourhood stands out.

The curse of dimensionality caveat: as you add features, distances between all points converge and "nearest" loses meaning - everything looks equidistant. Scale your features (distances are unit-sensitive) and keep the feature set lean. Distance methods are strongest in low-to-moderate dimensions.

Part 2 · Local Outlier Factor (Breunig et al., 2000)

Density, relative to your neighbours 5 min live

A global distance has one blind spot: real data is not uniformly dense. A point sitting in a naturally sparse region has a large kNN distance and looks anomalous - but it is perfectly normal for its neighbourhood. LOF fixes this by comparing each point's local density to the density of its neighbours. If yours is roughly the same as theirs, LOF ≈ 1 (normal); if yours is far lower, LOF ≫ 1 (anomalous). It is a ratio, so it adapts to whatever density each region happens to have.

🔍 Click to zoom - LOF adapts to local density where a global distance cannot

Self-studyReading negative_outlier_factor_2 min read▶

scikit-learn's LocalOutlierFactor stores its scores in negative_outlier_factor_ after fit_predict. A few gotchas:

The stored value is the negative LOF, so normal points sit near -1 and anomalies are more negative. Flip the sign to get the familiar LOF where higher = more anomalous.
By default it runs in outlier mode (fit and predict on the same data). For novelty detection - training on clean data, scoring new points - set novelty=True and use .predict().
contamination sets the flag rate. It is a precision/recall dial, not a fact about the world - pick it from your alert budget, not by wishful thinking.

Demo 1 of 2

Features + kNN distance ★ 12 min · everyone builds

Turn the one-dimensional Lumen series into a feature matrix, then score each day by its mean distance to its k nearest neighbours and check the top scores against the truth.

Build features: value, a one-day lag, and rolling mean and std over a week.

Scale the features, then compute each day's mean distance to its k nearest neighbours.

Flag the highest-scoring days and score precision, recall, and F1 against truth.

★ Run it now (Python) - kNN-distance detectorimport pandas as pd, numpy as np from sklearn.neighbors import NearestNeighbors from sklearn.preprocessing import StandardScaler # --- rebuild the labeled Lumen series (from b1) --- idx = pd.date_range("2026-01-01", periods=90, freq="D") season = np.array([30,42,48,40,22,-95,-110]); i = np.arange(90) y = 900 + 1.2*i + season[i % 7] + 10*np.sin(i*1.7) + 7*np.cos(i*0.6) y = y.astype(float) truth = [20, 45, 63, 70, 71, 72] y[20] += 430; y[45] -= 380 y[63] = y[63] - season[63 % 7] + season[6] - 40 y[[70,71,72]] += [175,190,170] s = pd.Series(np.round(y), index=idx, name="sessions") # --- build a small feature matrix --- df = pd.DataFrame({"value": s.values}) df["lag1"] = df["value"].shift(1) df["roll_mean"] = df["value"].rolling(7).mean() df["roll_std"] = df["value"].rolling(7).std() df = df.dropna() # drops the first 6 rows (no rolling yet) offset = len(s) - len(df) # = 6, to map rows back to day indices X = StandardScaler().fit_transform(df.values) # scale: distances are unit-sensitive def score(pos, name): f = set(int(p)+offset for p in pos); t = set(truth) tp = len(f & t); fp = len(f - t); fn = len(t - f) p = tp/(tp+fp) if tp+fp else 0; r = tp/(tp+fn) if tp+fn else 0 f1 = 2*p*r/(p+r) if p+r else 0 print(f"{name:5s} caught {tp}/{len(t)} P {p:.2f} R {r:.2f} F1 {f1:.2f} {sorted(f)}") k = 5 nn = NearestNeighbors(n_neighbors=k+1).fit(X) # +1: point 0 is itself dist, _ = nn.kneighbors(X) knn_score = dist[:, 1:].mean(axis=1) # mean distance to k neighbours top = np.argsort(knn_score)[-6:] # flag the 6 loneliest days score(top, "kNN") # kNN caught 4/6 P 0.67 R 0.67 F1 0.67 [20, 21, 45, 46, 71, 72]

Scaling is not optional Distances add up feature by feature, so an unscaled value in the hundreds would drown a roll_std in the tens - the detector would effectively see only one feature. StandardScaler puts every feature on the same footing. Forgetting it is the most common reason a distance detector "does not work".

Demo 2 of 2

LOF, and why local beats global ★ 10 min · build your own

Run LOF on the same feature matrix, then settle the argument: on a two-density example, watch a global kNN distance mis-flag a perfectly normal sparse point while LOF keeps it - and still catches the real outlier.

Fit LocalOutlierFactor on the Lumen features, read negative_outlier_factor_, flag, and score.

Build a toy dataset with a dense cluster and a sparse cluster plus one true outlier.

Compare a global kNN distance against LOF for the outlier and for a normal sparse-cluster point.

★ Run it now (Python) - LOF on the Lumen featuresfrom sklearn.neighbors import LocalOutlierFactor # (continues from Demo 1: X, offset, truth, score already defined) lof = LocalOutlierFactor(n_neighbors=20, contamination=0.06) pred = lof.fit_predict(X) # -1 = anomaly, 1 = normal lof_score = -lof.negative_outlier_factor_ # flip sign: higher = more anomalous score(np.where(pred == -1)[0], "LOF") # LOF caught 4/6 P 0.80 R 0.67 F1 0.73 [20, 21, 45, 71, 72] # on Lumen, LOF edges out the global kNN distance on precision (0.80 vs 0.67)

★ Run it now (Python) - global distance vs LOF on varying densityimport numpy as np from sklearn.neighbors import NearestNeighbors, LocalOutlierFactor rng = np.random.default_rng(7) dense = rng.normal([0, 0], 0.25, size=(60, 2)) # tight cluster sparse = rng.normal([5, 5], 1.10, size=(25, 2)) # loose cluster Xc = np.vstack([dense, sparse, [[2.5, 2.5]]]) # + one true outlier in the gap oi = len(Xc) - 1 # the outlier's index # global kNN distance nn = NearestNeighbors(n_neighbors=6).fit(Xc) d, _ = nn.kneighbors(Xc); knn = d[:, 1:].mean(1) sn = 60 + int(np.argmax(knn[60:85])) # loneliest NORMAL sparse point # LOF lof = LocalOutlierFactor(n_neighbors=15); lof.fit_predict(Xc) L = -lof.negative_outlier_factor_ print(f"global kNN: outlier {knn[oi]:.2f} vs sparse-normal {knn[sn]:.2f}") print(f"LOF: outlier {L[oi]:.2f} vs sparse-normal {L[sn]:.2f}") # global kNN: outlier 2.38 vs sparse-normal 1.99 -> barely different, mis-flags the normal # LOF: outlier 2.56 vs sparse-normal 1.36 -> clean separation (LOF~1 = normal)

Real world

This is exactly the trap in mixed populations. Power users click ten times more than casual users - a naturally "sparse" region of behaviour. A global distance rule flags every power user as anomalous and floods the queue with false alarms, the fastest route to a muted alert. LOF asks the fairer question - "is this account odd for accounts like it?" - and reserves the flag for the genuinely strange. When your data has sub-populations of different density, prefer LOF.

Homework

Try it yourself - this week ◐ 30-45 min total

Reproduce the kNN and LOF scores and confirm they match the commented output.
Sweep k (and n_neighbors) from 3 to 30. Watch precision and recall move - small k is jumpy, large k oversmooths.
Drop the rolling features and rerun with value + lag only. How much does the leaner feature set cost you, and why?
Add three noise features of pure randomness to the matrix and rerun. Watch the curse of dimensionality erode the scores.
Optional: read Breunig, Kriegel, Ng & Sander (2000), "LOF: Identifying Density-Based Local Outliers" - the paper that defined the local density ratio.

Source material

Sources covered (cited)

Taught from the density-based detection canon. This session covers:

✓Breunig, Kriegel, Ng & Sander (2000), "LOF: Identifying Density-Based Local Outliers," ACM SIGMODParts 1-2 · local density ratio, why global distance fails on varying density

◐scikit-learn "Novelty and Outlier Detection" user guideDemos · NearestNeighbors, LocalOutlierFactor, novelty vs outlier mode

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · What does the kNN-distance anomaly score measure?

Points deep in a cluster have small neighbour distances; a point out in empty space has a large one, so it rises to the top when you rank by the score.

2 · What does the Local Outlier Factor compare?

Because LOF is a density ratio, each region's own tightness becomes its baseline, so it flags a point only when it is sparse relative to its neighbours.

3 · When does LOF clearly beat a global kNN distance?

A naturally sparse-but-normal point has a large global distance and gets wrongly flagged; LOF's local ratio keeps it normal while still catching the true outlier.

Builder session 4 cheat sheet · pin this

kNN distancescore = distance to the k-th nearest neighbour (or mean of k). Big = isolated = anomalous.

LOFratio of a point's local density to its neighbours'. ≈ 1 normal, ≫ 1 anomalous (Breunig 2000).

Local beats globalon varying-density data, a global distance mis-flags sparse-but-normal points; LOF does not.

Multivariatethese methods shine with several features; on a raw 1-D series they collapse to a z-score.

Feature a series firstvalue + lag + rolling mean/std turn time into geometry the detectors can read.

Scale everythingdistances are unit-sensitive - StandardScaler before NearestNeighbors or LOF.

Curse of dimensionalitytoo many features and all distances converge; keep the set lean.

Next sessionb5 - Isolation Forest: isolate anomalies by random splits instead of measuring distance.