When clustering lies

• Lie 1 - noise still clusters. k-means partitions whatever you feed it. Ask for five clusters of pure random noise and you receive five clusters of pure random noise, colored and centered and ready for a deck. Today's lab is this lie, performed live.
• Lie 2 - assumption violations. Even on data with real structure, k-means assumes roughly round, similar-sized blobs. Feed it stretched (anisotropic) blobs, unequal variances, or the wrong k and it cuts confidently across the real boundaries. scikit-learn's own docs ship a demo of exactly this.
• Lie 3 - fake elbows. Session 4 demoted the elbow method; here is the reason at full strength. Even uniform random noise traces a bendy inertia curve, so "I see an elbow" is evidence of almost nothing (Schubert's 2023 critique: stop using the elbow criterion).
• Lie 4 - artifact segments. Sometimes the structure is real but manufactured by the pipeline: session 2's cents column invented wallet-band "segments" that were really a unit choice; session 4's k=12 sliced five real groups into twelve fundable-sounding shards. The data lied because we made it lie.

Lies 1-3 are the algorithm's doing; lie 4 is yours. That makes it the most common in practice, because it survives review: the clusters ARE separable, the silhouette IS decent, and nothing on screen reveals that the separation traces a billing-system unit or an arbitrary k rather than customer behavior. The only defenses are the ones this course drilled: know your units (session 2), let the data pick k (session 4), and profile segments on features you did not cluster on (below). An artifact segment is a pipeline shadow - move the lamp and it moves with it.

One honest paragraph before anyone gets comfortable: none of these is a proof. A clustering can pass every tell and still be a mediocre way to slice the business, or fail one tell for a fixable reason. They are evidence that accumulates - exactly the hierarchy session 4 taught for picking k, now applied to the bigger question of whether to trust the clustering at all.

The recipe is deliberately boring: match the shape (same number of rows and columns), match the ranges (uniform random within each feature's observed min-max, after the same scaling you applied to the real data), run the identical clustering with the identical k, and record the silhouette. Repeat a handful of times and keep the typical value. That number is your floor - the score confidence costs nothing to achieve. It is field practice rather than textbook doctrine, which is exactly why so few decks include it and why yours should.

Everything above is about scores; the plots have their own lie. Session 6 warned you and session 9 is where the warning belongs: t-SNE at low perplexity makes pure noise look clustered - tight, separated blobs, manufactured entirely by the projection (Distill's "How to Use t-SNE Effectively" demonstrates it interactively). And UMAP's own FAQ blesses clustering on its output only "with care," recommending density-based methods rather than k-means, because density is not preserved. The rule from session 6, restated for skeptics: projection pictures are for LOOKING, never for measuring - a beautiful 2D scatter of clusters is a hypothesis, not a tell. The scores above, with their baselines, are the tells.

• Continuous gradients. Customer value is often a smooth ramp, not clumps. Clustering a ramp manufactures boundaries that exist nowhere in the data; percentile tiers (top 10%, next 40%, rest) are honest, simple, and easier to maintain than fake clusters.
• Tiny samples. A few dozen rows cannot support stable clusters - every tell fails, every re-run reshuffles. Sort, eyeball, and use judgment; say so.
• One-dimensional structure. If one feature carries the story, sort by it. A clustering of six features that secretly follows one of them is a complicated way to hide a ranking.
• The decision is already made. If the campaigns were designed before the analysis, the segmentation is theater - it can only bless or be ignored. Decline politely; theater has no noise baseline.

And one for the survivors: drift. Real pipelines re-run, and segments move - customers lapse, freshen, upgrade. Schedule reprofiling, alert when segment migration spikes, and honor the review date you wrote on session 7's segment sheet. A segmentation without a review date is a lie with a delay.

How do you tell a gradient from clumps before running anything? Plot the histogram of the feature that matters most. Clumps show as multiple humps with gaps; a gradient shows as one smooth slope. Then run the tells: on a true gradient, k-means will still return your k groups (it always does), the silhouette will be modest, and above all the STABILITY check fails in a telltale way - the boundaries land somewhere different on every resample, because there is no gap for them to snap to. Moving boundaries on stable data is the gradient's fingerprint. When you see it, reach for percentile tiers and write the memo: they deliver the same operational value (differentiated treatment by level) without claiming groups that do not exist.

One question this session raises but does not answer: what about the points that belong to NO group - the fraud, the glitches, the one-in-a-thousand outliers? That is anomaly detection, the OTHER "find the weird stuff" toolkit and the second pillar of Ng's unsupervised week. It has its own full course on this shelf: learn-anomaly-detection-with-phoebe (the Lumen case). This course finds the groups; that one finds the exceptions. Together they are the two honest answers to "what is in this pile?"