Why not just number the categories 8 min live
The tempting shortcut is to map channel to 0 through 8 and move on. Google's crash course explains the damage in one sentence, and it is worth reading slowly: "the model would treat the indexed values as continuous floating-point numbers. The model would then consider 'purple' six times more likely than 'orange.'"
An index is not a label. It is a number with an order and a magnitude, and a linear model will happily fit a single weight to it - meaning your model now believes affiliate traffic is exactly nine times whatever email is. Nothing about your data said that. You said it, accidentally, when you chose the integers.
One-hot removes the arithmetic. Nine categories become nine binary columns, exactly one of which is 1 for any given row. No order, no magnitude, one weight learned per category independently.
sklearn's OneHotEncoder carries the parameters that turn this from a demo into something production-safe. These five are the ones you will actually set.
| Parameter | What it does | Why you set it |
|---|---|---|
handle_unknown | error · ignore · infrequent_if_exist | A category appears in production that was not in training. error crashes, ignore emits all zeros, infrequent_if_exist routes it to the rare bucket |
min_frequency | An integer count or a fraction of rows | Levels below the threshold are grouped into one infrequent category instead of getting their own barely-populated column |
max_categories | A hard ceiling on output columns | Caps the blast radius. Note the docs' fine print: "max_categories includes the feature that combines infrequent categories" |
drop | 'first' or 'if_binary', giving n-1 columns | "Useful to avoid co-linearity" in linear models, where the full set of dummies is perfectly redundant |
sparse_output | Return a sparse matrix or a dense array | Sparse is the default and the right choice at scale - and it is why b2's MaxAbsScaler advice exists |
LiveThe gotcha that costs an afternoon3 min▶
np.nan and None are treated as separate categories by OneHotEncoder. If your pipeline reads some rows from a CSV (missing values arrive as np.nan) and others from a JSON API (missing values arrive as None), you will silently produce two different "missing" columns describing the same absence, each with half the data and half the statistical power.
The fix is dull and reliable: normalize your nulls to one representation before the encoder ever sees them. Do it in the ingest step, not in the model code, so every downstream consumer inherits the same convention.
Missing is often signal, not absence. A Lumen session with no channel is usually direct traffic that failed attribution, which is a meaningfully different customer from one that clicked a paid ad. Treating that null as its own category is frequently the right call rather than a failure to impute - which is exactly the argument b7 makes at full length.
Self-studyWhen to drop a column, and when not to3 min read▶
drop='first' removes one dummy per feature, leaving n-1 columns. sklearn's stated reason is that it is "useful to avoid co-linearity in the input matrix" - with all n dummies present, any one of them is exactly one minus the sum of the others, and an unregularized linear model has no unique solution.
- Drop for plain linear or logistic regression when you intend to read the coefficients. Each remaining weight becomes "relative to the dropped baseline category", which is usually what you wanted to say anyway.
- Do not drop for tree-based models, where collinearity is not an issue and dropping a level just makes one category harder to isolate with a single split.
- Do not drop when you are using L1 or L2 regularization, which handles collinearity on its own and where the penalty applies asymmetrically once a baseline is implicit.
drop='if_binary'is the tidy middle ground: it drops only for two-level features, where the second column is pure redundancy no matter what model you use.
The high-cardinality problem 6 min live
One-hot works beautifully at nine levels. It stops working somewhere before a thousand, and Google names the failure directly: "Categorical data tends to produce high-dimensional feature vectors... High dimensionality increases training costs and makes training more difficult."
This course adds postal_code to the Lumen case precisely to make the problem concrete. It has roughly 1,200 levels across 40,000 rows. One-hot encoding it adds 1,200 columns, most of which are 1 in a few dozen rows and 0 in every other row. The model is now being asked to learn 1,200 weights from evidence that, for most of them, amounts to a handful of observations.
| Option | How it handles 1,200 levels | Cost |
|---|---|---|
| Rare grouping | min_frequency folds everything below a threshold into one infrequent column | Simple and safe. You lose the identity of every small area |
| Hashing | Hash each level into a fixed number of buckets, say 64 | No fitting and no vocabulary to store, but collisions are silent and irreversible |
| Target encoding | Replace each level with a smoothed statistic of the target for that level - one column total | Powerful. Leaks the target unless you cross-fit. This is Part 3 |
| Embeddings | Learn a dense low-dimensional vector per level | Best on very high cardinality with real structure. Needs more data and more machinery (b9) |
Self-studyRare grouping is underrated3 min read▶
Before reaching for anything clever, try the boring option and measure it. Setting min_frequency=0.01 on postal_code keeps only the areas that appear in at least 1% of sessions and folds the rest into a single bucket. Twelve hundred columns typically become a few dozen, and the model gets to learn each of those few dozen weights from a real sample rather than a rumour.
Two practical benefits people forget:
- It gives you a natural landing zone for unseen categories in production. Pair it with
handle_unknown="infrequent_if_exist"and a postal code that never appeared in training is routed to the infrequent bucket rather than crashing or vanishing into all-zeros. - It is fully reversible and fully explainable. You can show a stakeholder exactly which levels were merged. Neither hashing nor an embedding lets you do that.
Target encoding, done right 10 min live
Target encoding replaces a category with a statistic of the target for that category: postal code 048624 becomes the conversion rate of sessions from 048624. One column instead of 1,200. The idea is obvious and the implementation is full of trapdoors, which is why sklearn documents the maths rather than hiding it.
For a binary target, each level i is encoded as a blend of its own rate and the global rate:
Read lambda_i as a confidence dial. A category with thousands of rows has lambda_i close to 1, so it is encoded as its own rate. A category with three rows has lambda_i close to 0, so it is dragged back toward the global mean where it can do less damage. sklearn states the effect in one line: "Large smoothing factors put more weight on the global mean."
sklearn's own demonstration of the dial. On the same three categories, smooth=5000 produces encodings of [44.1, 44.4, 44.3] - the categories have been smoothed into near-identity with the global mean, and the feature carries almost nothing. smooth=1.0 on the identical data produces [21, 80.8, 43.2] - the raw category rates, barely regularized, and now trusting three separate small samples completely. Neither extreme is the answer; the parameter is the whole game.
Now the part that matters more than the formula. If you compute each category's rate from the same rows you then train on, every row has contributed to its own encoding. The feature contains a whisper of the answer, and the model hears it.
fit_transform, the training data is split into k folds (determined by the cv parameter) and each fold is encoded using the encodings learnt using the other k-1 folds." The stated purpose: to prevent target information from leaking into the train-time representation, especially for non-informative high-cardinality categorical variables.
Which produces the single most surprising line in the sklearn preprocessing docs, and the one to say out loud in your next code review: fit(X, y).transform(X) does not equal fit_transform(X, y). These are different operations by design. fit_transform cross-fits; fit followed by transform does not, and the docs explicitly discourage using that pair on training data.
Self-studyWhat the peer-reviewed evidence says3 min read▶
Target encoding has a reputation as a Kaggle trick, which undersells it. Pargent, Pfisterer, Thomas and Bischl (2021) benchmarked encoding strategies for high-cardinality categorical features across many datasets and models, and found that "regularized versions of target encoding... consistently provided the best results" compared with integer and one-hot encoding.
Note the word doing the work: regularized. The finding is not "target encoding wins". It is "target encoding with the smoothing and the cross-fitting wins". The unregularized version is the thing you are about to watch fail in the workbench, and the paper is not endorsing it.
This is also the honest answer to "should I always use target encoding?" - no. Use one-hot with rare grouping when cardinality is modest and you want interpretability. Reach for a regularized target encoder when one-hot would inflate the feature space beyond what your row count can support.
Watch a useless column score 0.931 ★ 14 min · everyone runs it
The workbench below target-encodes Lumen's postal_code two ways on the same rows. Read the setup carefully before you press anything, because the setup is the whole point.
The setup. postal_code has 1,200 levels and is drawn independently of the target. It knows nothing. There is no signal in it to find, by construction - like sklearn's own near_unique demo column.
Naive encoding - fit the per-level rates on all training rows, then reuse them on those same rows: training AUC 0.931, held-out 0.493. A column containing no information just produced a training score you would celebrate.
Cross-fitted - each fold encoded using the encodings learnt from the other folds, which is what TargetEncoder.fit_transform does: training 0.509, held-out 0.493. The held-out number did not move. The training number told the truth for the first time.
Sit with the gap. Naive: 0.931 train against 0.493 held-out. Cross-fitted: 0.509 against 0.493. Cross-fitting did not improve the model - it improved your ability to see that there was never a model here at all.
sklearn's own numbers, at scale. On 50,000 rows with a near-unique category column, their documented example reports test R² of 0.634 without cross-fitting versus 0.793 with it. Their docs call the first case "catastrophic overfitting" - not sloppy, not suboptimal. Catastrophic.
Try it yourself - this week ◐ 25-35 min total
- Count the distinct levels in every categorical column you own. Anything above about fifty is a candidate for rare grouping; anything above a few hundred is a candidate for target encoding or embeddings.
- Take your highest-cardinality column and one-hot it with
min_frequencyat 0.01, then 0.005, then 0.001. Record how many columns each produces and how the held-out score moves. Usually less than you expect. - Reproduce the trap deliberately: target-encode a column with
fitthentransformon training rows, and compare the training score with the same encoder inside aPipeline. Seeing your own number inflate is worth more than reading about mine. - Audit your codebase for
TargetEncoder,LabelEncoderor any hand-rolled group-mean encoding used outside a Pipeline. Each one is a leak waiting to be found by someone else.
Official sources covered
Tonight covers Google's categorical-data module for the reasoning and sklearn's encoder documentation for the mechanics, plus the peer-reviewed benchmark that backs the regularized approach. Certificates and graded exercises stay with the official providers. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Why is mapping channel to the integers 0 through 8 the wrong move?
Google's example says it exactly: the model "would then consider 'purple' six times more likely than 'orange.'" You did not observe that ordering in the data - you invented it when you picked the integers.
2 · What does cross-fitting inside TargetEncoder.fit_transform actually prevent?
The training data is split into k folds and "each fold is encoded using the encodings learnt using the other k-1 folds". Without it, a column with no signal at all scored a training AUC of 0.931 in tonight's workbench, and sklearn's own 50,000-row example dropped from test R² 0.793 to 0.634 - which their docs call "catastrophic overfitting".
3 · When does target encoding beat one-hot encoding?
Rules of ML #21 sets the budget: learnable weights are "roughly proportional to the amount of data you have". Pargent et al. found "regularized versions of target encoding... consistently provided the best results" - regularized being the operative word. At nine levels, one-hot with rare grouping is simpler and more explainable.