Regression + ranking metrics

sklearn's evaluation guide compresses this whole zoo into one decision rule - what are you actually trying to predict?

• Predicting a mean - score with squared error (RMSE family).
• Predicting a median - score with absolute error (MAE family).
• Predicting a quantile (p90 demand for safety stock) - score with pinball loss.
• Predicting a probability - score with proper scoring rules (Brier, log loss - b3's territory).

The doctrine cuts both ways: a model trained toward the mean but reported on MAE is being graded against a target it never aimed at. Say what the model predicts, then the metric picks itself.

The intro-ml course's order-value model reached R² ~0.45 with multiple features - and that is honest work on noisy humans. Individual purchase behavior is mostly unexplainable variance; nobody's features capture mood, payday, or a friend's recommendation. Two habits follow:

• Calibrate expectations by domain. R² 0.45 on human behavior can be excellent; R² 0.98 on a physical sensor can be mediocre. There is no universal "good R²."
• Never read R² alone. High R² with structured residuals is common - the model can explain the variance and still be systematically wrong for a whole segment (every order above $200, say). Residual plots are part of the metric, not an optional extra.

The intro-ml course covers the mechanics hands-on; here is the whole family as a refresher, with the baseline wired in:

import numpy as np
from sklearn.metrics import (
    mean_absolute_error, root_mean_squared_error,
    mean_absolute_percentage_error, r2_score,
    median_absolute_error, mean_pinball_loss,
)

mae  = mean_absolute_error(y_test, pred)
rmse = root_mean_squared_error(y_test, pred)
mape = mean_absolute_percentage_error(y_test, pred)  # guard zeros first
r2   = r2_score(y_test, pred)
med  = median_absolute_error(y_test, pred)
p90  = mean_pinball_loss(y_test, pred_q90, alpha=0.9)

# the lie detector: the predict-the-mean baseline
baseline = np.full_like(y_test, y_test.mean(), dtype=float)
mae_base = mean_absolute_error(y_test, baseline)

One habit worth stealing: print every metric as a pair - model vs baseline - in the same log line. Reviews go faster when the lie detector is already in the output.

• precision@k - of the top k items, how many are relevant? The metric of every budget cutoff: "we can act on 500, how many are real?"
• recall@k - of all relevant items, how many made the top k? The metric of coverage: "how many real buyers does the top 500 even contain?"
• NDCG - handles GRADED relevance (a perfect match beats an okay match), discounts each position by log2 of its rank so the top of the list dominates, and normalizes to 0-1 against the ideal ordering. In sklearn: ndcg_score with k= for a cutoff. Canonical citation: Jarvelin & Kekalainen 2002.
• average_precision_score - sklearn's AP summarizes the whole precision-recall tradeoff of the ranking. It is deliberately step-wise, not trapezoidal: linear interpolation of PR curves is overly optimistic (the b3 warning, still true here).

Honest note: true IR-style MAP@k - averaged over queries the way search papers report it - is not in sklearn. If a vendor quotes MAP@10, ask exactly how they computed it.

Here is how ranking metrics enter Lumen's story: rank all sessions by the conversion model's score, and marketing's outreach budget covers the top 500. Nobody below position 500 gets touched, so the quality of the whole ranking is irrelevant - precision@500 is not a proxy for the business metric, it IS the business metric: real converters per 500 outreach dollars spent. This is the rare, happy case where the metric and the money are the same number.

import numpy as np
from sklearn.metrics import ndcg_score

# graded relevance (0-3) vs model scores, top-heavy by design
ndcg = ndcg_score([relevance], [scores], k=500)

# precision@k by hand: slice the top k, count the hits
k = 500
top_k = np.argsort(scores)[::-1][:k]
precision_at_k = y_true[top_k].sum() / k

NDCG intimidates people because the formula has three moving parts. Read them one at a time and it is friendly:

• Graded relevance. Items are not just relevant/irrelevant - a 3 (perfect) beats a 1 (marginal). Binary labels still work; grades are the upgrade.
• The log2 rank discount. Position 1 counts full weight, position 2 divided by log2(3), and so on - relevance buried at rank 40 is worth almost nothing. The discount encodes the belief that users read from the top.
• The N is normalization. The score is divided by the best achievable ordering of the same items, so 1.0 always means "the ideal ranking" and scores compare across queries.

Two reading rules: an NDCG without a stated k is a whole-list score (rarely what the business does), and an NDCG compared across DIFFERENT datasets means little - the normalizer changes with the items. Compare rankers on the same data, at your real k.

• Money target? Start with MAE and RMSE together, then pick by outlier pain: if one huge miss breaks something (inventory, capacity), lead with RMSE; if you want the honest typical miss, lead with MAE (or median absolute error when the tail is pure noise).
• Comparing across scales? (stores, products, currencies) - MAPE, with a zero-guard: exclude or floor near-zero actuals first, and remember it penalizes over-forecast less.
• Ranked list with a budget cutoff? - precision@k at YOUR k. The cutoff is a business fact, not a modeling choice.
• Graded relevance, order matters everywhere? - NDCG (with k if only the top matters).

And in every branch: quote the number NEXT TO its dumb baseline (predict-the-mean, or a random ranking). A metric without a baseline is a mood.