Selection + importance

Look back at the ladder you climbed in b1. Selection is the last rung, and it is the smallest one on the board:

• + Interactions: held-out AUC 0.812
• + Selection: held-out AUC 0.813

One thousandth. That is the accuracy case for feature selection on this dataset, and stating it plainly is the point - most courses quietly imply selection is a performance technique. It usually is not. What it actually buys is worth more than a decimal place:

• Simplicity. Fewer columns to explain, review, and defend when someone asks why the model did that.
• Stability. Fewer weights spread across noise means less variance between retrains, and less sensitivity to a shift in a column nobody is watching.
• Serving cost. Every feature in the model is a feature that must be computed, joined, monitored and kept alive at prediction time. A column you dropped is a pipeline you never have to fix at 2am.

Frame it that way with your team and the conversation improves immediately: you stop arguing about a thousandth of AUC and start arguing about how many things you want to keep running.

Rule 22 is a maintenance rule, not a modelling one. An unused feature still has an owner, an upstream job, a schema, a backfill, and an on-call rotation attached to it. It still breaks. It still shows up in the incident channel. And because nobody remembers whether the model uses it, nobody dares delete it.

The cheapest moment to remove a feature is the moment you can show it earns nothing. That is what the rest of this session is about: producing evidence solid enough that the deletion happens.

Kaggle Learn's definition is the one to carry around: mutual information is "a measure of the extent to which knowledge of one quantity reduces uncertainty about the other." It is the first thing to reach for on a new dataset, because it answers "which columns are worth my attention this afternoon" without fitting anything.

It beats correlation for a specific reason, stated plainly in the same lesson: "correlation only detects linear relationships." A U-shaped relationship, a threshold effect, a category that matters only at the extremes - correlation reports roughly zero for all of them. Mutual information does not.

The scale: minimum 0.0, meaning the two quantities are independent, and no fixed maximum - though "values above 2.0 or so are uncommon", so treat a score above 2 as a reason to check for leakage rather than a reason to celebrate.

The first caveat is the one that bites in this course specifically. In b6 you built interactions - relationships that no single column carries. A univariate metric is structurally blind to exactly those. If you had used mutual information to prune before building interactions, you would have deleted the ingredients.

• Classification: chi2 (non-negative features only), f_classif (linear dependency), mutual_info_classif (any dependency, slower).
• Regression: f_regression, mutual_info_regression.
• The mistake sklearn bothers to warn about: crossing the two lists. "Beware not to use a regression scoring function with a classification problem, you will get useless results." Nothing raises an error - you simply get a ranking that means nothing, which is worse.

sklearn's warning, verbatim, is worth reading in full because every clause does work:

Impurity-based importances "favor high cardinality features (typically numerical features) over low cardinality features such as binary features or categorical variables with a small number of possible categories", and "can give high importance to features that may not be predictive on unseen data when the model is overfitting."

The mechanism is simple once you see it. A column with thousands of distinct values offers thousands of possible split points, so it will almost always find some split that reduces impurity on the training rows. It looks powerful because it is flexible, not because it is predictive. Meanwhile a genuinely useful binary flag gets one split point and one shot, and finishes lower down the list.

That is why a customer id, a raw timestamp, or a high-cardinality postal code so often floats to the top of an importance chart that then gets pasted into a slide.

The alternative, in sklearn's words: permutation importance "avoids this issue, since it can be computed on unseen data." Shuffle one column, re-score, and measure how much the score falls. If the model was genuinely relying on the column, breaking it hurts. If the model was memorising, breaking it changes nothing on rows it never saw.

Permutation importance can be computed on training data or on held-out data, and comparing the two is a diagnostic in its own right. sklearn states the rule verbatim: "Features that are important on the training set but not on the held-out set might cause the model to overfit."

from sklearn.inspection import permutation_importance

r = permutation_importance(
    pipe, X_test, y_test,            # held-out rows, never the training rows
    n_repeats=30, random_state=42, scoring="roc_auc")

for i in r.importances_mean.argsort()[::-1]:
    if r.importances_mean[i] - 2 * r.importances_std[i] > 0:
        print(f"{feature_names[i]:<24} "
              f"{r.importances_mean[i]:.3f} +/- {r.importances_std[i]:.3f}")

n_repeats is what turns a single shuffle into an estimate with a spread. Without it you cannot tell a real drop from a lucky permutation, and the filter above - keep only features whose mean importance clears two standard deviations - is the cheapest way to stop yourself reading noise as a ranking.

Permutation importance has one failure mode you will meet on real tabular data, and it is easy to walk into. If two features are strongly correlated, permuting one leaves the model its correlate to fall back on. The score barely moves. Do the same to the other, and the score barely moves again. Both report low importance, and both are actually load-bearing.

Drop them both on that evidence and the score collapses, which is a confusing afternoon.

The fix is mechanical: cluster the correlated features first - a hierarchical clustering on the correlation matrix does it - and keep one representative per cluster before you interpret any importance number. You are then measuring the cluster's contribution, which is the thing you actually wanted to know.

1. Split first, before anything else touches the data.
2. Explore with mutual information on the training rows only - to decide where to spend your afternoon, not to prune.
3. Build features, including the interactions from b6, before any univariate pruning. Univariate metrics cannot see combinations, so pruning first deletes the ingredients.
4. Put the selector inside the Pipeline, so it re-fits within every cross-validation fold.
5. Judge with permutation importance on held-out data, with n_repeats, after clustering correlated features.
6. Delete the column for real - Rule 22 - and delete the pipeline that produced it too, or you have kept the cost and lost the benefit.