Missing data as signal

Strip away the statistics vocabulary and there are three stories a blank cell can be telling you:

• Missing completely at random. The blankness has nothing to do with the row. A connection dropped, a batch job half-failed, a device rebooted mid-form. The rows with gaps look like the rows without them, so filling the gap with a central value distorts very little.
• Missing at random, given other columns. The blankness is not random, but it is fully explained by columns you already have. Lumen's mobile checkout never rendered the optional field, so device = "mobile" predicts the gap perfectly. You cannot ignore it, but you can model it from the columns that explain it.
• Missing NOT at random. The blankness depends on the missing value itself, or on the outcome. income is left blank disproportionately by high earners. support_note is empty precisely because the order went perfectly - which is exactly the kind of order you may be trying to predict. Here the missingness is the information.

You cannot prove which mechanism you are in from the data alone. You get there by asking the person who owns the form, the app screen, or the upstream job - which is why this session is as much an interview technique as a coding one.

Two separate damages, and teams usually only notice the second one much later:

• You destroy real signal. If blank means "this order was uneventful", then every blank row carried a usable fact about the target. Replace them all with the column mean and every one of those rows now says the same neutral thing. The feature that was there is gone, and no amount of model tuning brings it back.
• You bias the distribution. Filling 30% of a column with its own mean piles a third of the mass onto a single point. Variance shrinks, correlations get diluted toward zero, and any downstream scaler or binner you fit afterwards is fitting a shape the real world never had.

Both damages are silent. Nothing raises an error, the pipeline runs, and the score is merely a little worse than it should have been - the hardest kind of problem to notice.

Two defaults worth making automatic:

• Median, not mean, for money columns. Lumen's prior_30d_spend is right-skewed like every spend column you will ever meet - a long tail of big spenders drags the mean well above the typical customer. Impute with the mean and every blank row is quietly labelled a slightly-above-average spender. The median puts them where most customers actually are.
• A constant plus an explicit sentinel for categoricals. SimpleImputer(strategy="constant", fill_value="unknown") makes "we did not have this" a named level rather than a silent merge into whatever happened to be most frequent. The most_frequent strategy has its uses, but it hides the gap inside a real category, and you can never separate them again downstream.

Numerics get a median plus an indicator; categoricals get a named sentinel and then one-hot encoding that tolerates unseen levels at prediction time. Note that nothing is fitted outside the Pipeline:

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler
from sklearn.linear_model import LogisticRegression

numeric = Pipeline([
    ("impute", SimpleImputer(strategy="median", add_indicator=True)),
    ("scale", StandardScaler()),
])

categorical = Pipeline([
    ("impute", SimpleImputer(strategy="constant", fill_value="unknown")),
    ("encode", OneHotEncoder(handle_unknown="ignore")),
])

prep = ColumnTransformer([
    ("num", numeric, ["prior_30d_spend", "pages_viewed"]),
    ("cat", categorical, ["channel", "device", "product_category"]),
])

pipe = Pipeline([
    ("prep", prep),
    ("model", LogisticRegression(max_iter=1000)),
])

pipe.fit(X_train, y_train)   # every median learned from train rows only

One argument, add_indicator=True, is the whole missingness-as-signal lesson in code. The imputed value keeps the row usable; the indicator keeps the fact that it was blank.

IterativeImputer models each column with gaps from the other columns and cycles until the estimates settle. It is genuinely useful in the missing-at-random case, and it is still an experimental feature in sklearn - you must import the experimental enabler before the class is available at all, and the API can change between releases.

Two practical cautions before you reach for it: it is far more expensive than a median, and it makes the imputed values a function of the whole feature set, which means a single upstream column changing shape now moves values in a column you were not touching. Start simple, and let a measured gain justify the complexity - not the other way round.

• A column gets deprecated upstream. The producing service stops writing it, or writes it only for a subset of traffic. The column still exists, the schema still validates, and the values simply stop arriving.
• A form field is made optional. The most common cause and the most invisible one, because it is a product decision made in a different team's sprint, for perfectly good conversion reasons. Overnight, a column that was 2% missing becomes 60% missing.
• The mechanism itself changes. Worse than a rate change: a column that was missing completely at random becomes missing not at random, or the reverse. The correct handling flips, and your pipeline carries on doing the old thing.

The missingness rate of every feature is worth monitoring in its own right, on the same dashboard as your model's score. It is cheap to compute, it needs no labels, and it moves before your metrics do - a rate that jumps from 8% to 90% is visible on day one, while the business impact shows up weeks later.

Two thresholds worth setting per feature: an absolute one ("alert if more than 40% blank") and a relative one ("alert if the rate doubles versus the training window"). The relative one catches the slow drift the absolute one sleeps through.

This is where the practitioner track meets the leader track: in a3, missingness is framed as part of the feature budget - a data-quality signal that decides which features you can responsibly build at all. If a column cannot be relied on to arrive, its predictive power on historical data is a promise you cannot keep.