Support vector machines - the widest street

After training, the SVM keeps only the support vectors - the curb-touching points - and throws the rest of the training set away. On a clean problem that can mean keeping dozens of points out of thousands. This is why session 1's zoo map gave the SVM "a foot in both camps": it learns a boundary like a model-based method, but that boundary is defined by remembered training points, like an instance-based method.

The practical payoff is memory efficiency: unlike kNN, which ships the whole training set to production, a fitted SVM ships only its support vectors. The conceptual payoff is bigger - it tells you what the SVM believes: the boundary should be decided by the hardest, closest cases, not by the easy points deep inside each class.

C prices margin violations, and the scikit-learn guide gives the cleanest reading of it. Low C: violations are cheap, so the model accepts a wide, tolerant street - a smoother, more regularized boundary that may misclassify a few training points on purpose. High C: violations are expensive, so the model narrows the street until it fits every training point it can - and starts bending around noise. High C is the SVM's overfitting mode.

The sklearn practical-use tip worth memorizing: if your data is noisy, decrease C. Noisy labels plus a strict street means the street gets dragged around by mislabeled points. Lumen's ~3% conversion signal is exactly this kind of noisy, overlapping data - which is a hint about how today's demo will go.

Notice the pattern from the whole course: C is this animal's complexity knob, exactly as degree was for polynomials, k for kNN, and depth for trees. Same U-curve, new dial.

StatQuest's SVM Part 1 tells the story in two beats. Beat one: the maximal margin classifier - the widest street with zero violations allowed - is lovely on clean data and terrible in practice, because one outlier standing near the other class drags the entire street toward it. A single weird point should not redraw your boundary.

Beat two: allow violations, price them, and pick the width that cross-validates best - the soft margin. That is the actual "support vector classifier" everyone uses. Watch Parts 1-3 after the session; the margin story, the polynomial kernel, and the RBF kernel each get their own episode, and Josh's hand-drawn streets are the best in the business.

Same boundary explorer as sessions 1, 3, 5 and 7 - now with the RBF kernel. Drag gamma on the moons and watch the boundary go from almost-linear (small gamma, long reach, smooth) to wrapping individual points (large gamma, short reach, memorization).

Then switch to Lumen real and notice how little the bending buys you on an overlapping cloud - a preview of the demo's honest ending.

Gamma is a reach dial. Small gamma: each support vector's influence stretches far, many points blend into one smooth opinion, the boundary stays calm. Large gamma: influence dies off within a whisker, each point becomes its own island of opinion, and the boundary wiggles around individual training points - the RBF's overfitting mode, the exact cousin of k=1 and depth=None.

C and gamma overlap. Both control model complexity, and their effects interact - a strict street with long-reach bumps can behave like a tolerant street with short-reach bumps. The sklearn guidance: tune them together, on an exponential grid (powers of 10, then refine), never one at a time on a linear grid. In session 9 the fair harness does exactly this with GridSearchCV.

scikit-learn ships four built-in kernels for SVC, and one line covers each:

Default to linear-first thinking on tabular data, rbf when you have reason to believe the boundary curves. Poly and sigmoid are for when you can articulate why.

The sklearn docs say it plainly: support vector machines are "not scale invariant" and scaling is "highly recommended". Margins are measured in feature units - if prior_30d_spend runs 0-500 while pages_viewed runs 1-20, the street's width is effectively measured in spend and pages barely counts. You validated the same failure on kNN in session 5 (0.64 scaled vs 0.58 unscaled AUC); distance-based models share this Achilles heel.

The idiom to make it un-forgettable: make_pipeline(StandardScaler(), SVC()) - the scaler lives inside the pipeline, fit on train only, forever. There is no legitimate SVM workflow without it.

Kernel SVC (libsvm under the hood) solves a quadratic program whose cost scales roughly O(n²) to O(n³) with the number of rows. Double the data, pay four to eight times the compute. On Lumen's 30,000 training rows a default RBF fit stops being interactive - so today's demo subsamples 6,000 rows for the RBF round and says so out loud. Quietly subsampling is how leaderboards lie; loudly subsampling is engineering.

LinearSVC is the escape hatch. It uses liblinear, not libsvm - no kernel machinery, near-linear scaling, happy on millions of rows. When the boundary you want is straight, you get the full 30k rows AND a faster fit. That asymmetry (linear = all the data, kernel = a sample) is itself a fairness caveat to keep in mind when you read today's results.

SVMs do not natively produce probabilities - they produce signed distances to the street (decision_function). probability=True bolts probabilities on by running an expensive internal cross-validation (Platt scaling) after the fit, and the sklearn docs warn the results can disagree with decision_function - the argmax of the probabilities can differ from the side of the street the point is on.

It is also on its way out: deprecated in sklearn 1.9, removed in 1.11. For ranking metrics like ROC-AUC you never needed it anyway - decision_function scores rank points exactly as well, for free. That is what today's demo uses.

LinearSVC is almost-but-not-exactly a linear-kernel SVC: it penalizes the intercept along with the weights (SVC does not), and it optimizes squared hinge loss by default. In practice the differences are small on scaled data, but when the two disagree, this is why - worth knowing before you file a "bug".

For imbalanced targets like Lumen's ~3% conversion, class_weight="balanced" re-weights the margin violations so the rare class's mistakes cost more - the SVM family's version of the imbalance handling you met on logistic regression. Try it in the homework grid and see whether it moves the needle.