The zoo's probability engine
Naive Bayes is the only animal in the zoo that treats classification as pure probability bookkeeping. It never optimizes anything: no loss function, no gradient, no search. It reads the training data once, tallies a few per-class statistics, and every prediction after that is a multiply-and-compare. That makes it the fastest model you will ever fit - and a strangely honest one: everything it believes, you can print and inspect. The catch comes in two parts: an independence assumption that is almost always false, and output "probabilities" that rank well but calibrate terribly.
Bayes' theorem as a classifier 8 min live
The question we want is hard: P(convert | features) - given this session's channel, device, spend, will it convert? But the reverse question is easy: given a converter, what do their features look like? That is just counting rows in the training set. Bayes' theorem is the machine that turns the easy direction into the hard one: P(convert | features) ∝ P(features | convert) x P(convert). Score every class as prior times likelihood, predict the class with the biggest score. That argmax is the whole classifier.
LivePrior x likelihood, pick the winner5 min▶
For a new Lumen session, compute two scores. Score(convert) = P(convert) x P(these features | convert). Score(no) = P(no) x P(these features | no). Predict whichever is bigger. That is called the MAP decision - maximum a posteriori - and it is literally all NB does at prediction time.
Notice what we skipped: the denominator P(features) from the full theorem. It is the same for both classes, so it cannot change which score wins - we drop it, which is why the formula reads "∝" (proportional to) rather than "=". This shortcut is also the first clue about why NB's probabilities go wrong: it works with unnormalized scores and only rescales at the end.
The prior does real work here. Lumen converts at roughly 3%, so the "no" class starts with a ~30x head start; the likelihoods have to shout to overcome it. On imbalanced data, NB's prior is a built-in base-rate reminder - one thing it gets right by construction.
LiveBayes' theorem itself, in one breath3 min▶
Recap, not re-teach: Bayes' theorem says P(A | B) = P(B | A) x P(A) / P(B) - it converts "probability of evidence given hypothesis" into "probability of hypothesis given evidence". If you can state that sentence comfortably, keep going. If you want the derivation, the medical-test worked example, and the base-rate fallacy it protects you from, that lives in learn-statistics-with-phoebe ↗ - this course borrows the theorem and spends its minutes on the classifier built from it.
The naive bet 8 min live
There is a hole in Part 1: the likelihood P(features | convert) means the probability of this exact combination - this channel AND this device AND this spend AND these pages, jointly. With 7 features, the combinations explode; almost no exact combination appears in the training data even once. You cannot count what never occurs. So NB makes its famous bet: assume every feature is independent of every other, given the class. Then the joint likelihood factorizes into a product of per-feature likelihoods - P(email | convert) x P(mobile | convert) x P(spend | convert) x ... - and each factor is trivially countable on its own.
LiveThe assumption is false on Lumen - and NB scores ~0.66 anyway3 min▶
Is the naive bet true for Lumen? Not even close. Channel correlates with new_vs_returning (email lands disproportionately on returning customers), so "email" and "returning" are not independent given the class - NB double-counts their shared evidence. The assumption is simply wrong.
And yet: GaussianNB lands ~0.66 ROC-AUC on the full feature set - level with the depth-5 decision tree, a whisker under logistic's ~0.69. The lesson generalizes far beyond NB: a model with wrong assumptions can still rank well. Double-counting inflates the scores, but it often inflates them in a way that preserves their ordering - and AUC only measures ordering.
LiveThe hard rule: a decent classifier, a bad estimator3 min▶
The same double-counting that leaves rankings intact wrecks the probability values. sklearn's own user guide (1.9) puts it bluntly: naive Bayes is a decent classifier but a bad estimator - the outputs of predict_proba are "not to be taken too seriously". Piling up correlated evidence pushes scores toward extremes, so NB says "99.7% will convert" about sessions where the honest answer is "maybe 8%".
Operationally: ranking yes, calibrated probability no. Sorting sessions by NB score to pick the top 500 for an email push - fine. Feeding NB's 99.7% into an expected-revenue calculation or a bid price - a silent disaster. If you need real probabilities from NB, it must be recalibrated (a model-evaluation-course story; today, just never trust the raw values).
Self-studyWhy the naive bet relieves the curse of dimensionality3 min▶
Session 5's villain was the curse of dimensionality: in high dimensions, kNN needs enormous data because it must populate the whole joint space with neighbors. The naive bet is an escape hatch: NB never models the joint space at all. Each feature gets its own little 1-D distribution, estimated independently - a mean and a variance here, a handful of category counts there.
Estimating seven 1-D distributions takes dramatically less data than estimating one 7-D distribution. That is why NB shines exactly where kNN suffers: many features, few rows. Both StatQuest's NB episodes and the sklearn guide hit this beat - the decoupling of per-feature estimates is the reason NB works at all on wide data like text, where "features" are thousands of word counts.
Match the distribution to the feature 9 min live
"Naive Bayes" is a family, not a model. The engine - prior x per-feature likelihoods, pick the argmax - never changes. What changes is the shape you assume for each per-feature likelihood, and sklearn ships one variant per shape. Choosing the variant IS the modeling decision:
| Variant | Feature type | What it learns per class | Lumen / classic use |
|---|---|---|---|
GaussianNB | Continuous numbers | A mean and a variance per feature - one bell per class | Our spend / pages / hour columns |
MultinomialNB | Counts | Category frequencies, smoothed by alpha (Laplace) | THE text classic - word counts |
BernoulliNB | Binary presence / absence | P(feature on | class) - and it penalizes non-occurrence too | Our one-hot categoricals; binarize thresholds raw inputs |
ComplementNB | Counts, imbalanced | Statistics of the complement of each class | Imbalanced text - often beats MultinomialNB there |
CategoricalNB | Ordinal-encoded categories | A frequency table per category value | Categoricals without one-hotting |
LiveSmoothing - why alpha exists3 min▶
NB multiplies. Multiplication has a fatal failure mode: one zero kills everything. If the word "refund" never appeared in a training spam email, then P(refund | spam) = 0 by counting - and any future email containing "refund" gets Score(spam) = 0, no matter how much other evidence screams spam. One unseen feature vetoes the entire product.
alpha is the fix: add a pseudocount to every count before computing frequencies (StatQuest's word for it - pseudocounts). alpha=1 is classic Laplace / add-one smoothing, sklearn's default. Nothing is ever exactly zero, so nothing can veto. Smaller alpha trusts the counts more; bigger alpha flattens the differences. It is MultinomialNB's and BernoulliNB's one knob worth tuning.
Self-studyThe spam story - NB's greatest hit3 min▶
Lumen has no text columns, so we teach the spam pipeline conceptually and keep our hands on GaussianNB and BernoulliNB. When you meet text for real (a vectorizer feeding MultinomialNB), the "covered" section below names this gap honestly.
Self-studyToo much data for RAM? partial_fit1 min▶
Because NB's "model" is just running counts and running means, it can learn incrementally: model.partial_fit(X_chunk, y_chunk, classes=[0, 1]) in a loop over chunks streams through data that never fits in memory. One line to remember: NB is one of the few zoo animals with out-of-core training built in.
Build: the millisecond baseline ★ 18 min · everyone builds
Three experiments: GaussianNB on the full feature set, BernoulliNB on the one-hot categoricals alone, and a timing race against logistic regression. Then one plot - the predict_proba histogram - to see the bad-estimator warning with your own eyes.
Fit GaussianNB on everything. Scaled numerics plus one-hots through the usual pipeline (note sparse_output=False - GaussianNB wants a dense matrix). Expect ~0.66 ROC-AUC.
Fit BernoulliNB on the one-hot categoricals only. Binary features are its native food. Expect ~0.65 - four categorical columns carry nearly the whole GaussianNB signal.
Race it against logistic. Time both fits with time.perf_counter(). NB tallies counts in one pass; logistic iterates. On this data expect NB to be roughly an order of magnitude faster to fit.
Look at the probabilities. Histogram predict_proba. The overconfident clumps near 0 and 1 are the bad-estimator warning made visible.
Before session 7 ◐ 25 min
- Swap
ComplementNBin for BernoulliNB on the one-hot categoricals (it wants non-negative inputs - one-hots qualify). Does the imbalanced-data specialist move the AUC on Lumen's ~3% target? - Refit GaussianNB on the unscaled numerics. It works just as well - write one sentence on why (hint: each feature gets its own per-class mean and variance, so no distance or gradient ever compares one feature's scale to another's).
- Two sentences: name a situation where you would reach for NB before logistic regression, and the one output you would refuse to use downstream.
Three questions before you go 🎯 ◐ 90 seconds
1 · What exactly is "naive" about naive Bayes?
The bet is conditional independence given the class. It is almost always false (on Lumen, channel correlates with returning) - yet it turns an impossible joint estimation into countable 1-D pieces, and the rankings often survive the lie.
2 · Why should you never trust naive Bayes' predict_proba values?
The independence violation preserves the argmax far more often than it preserves the values. Use NB scores to sort and select; never feed them into expected-value math without recalibration.
3 · You have word-count features. Which variant, and why does its alpha parameter exist?
Counts belong to MultinomialNB. Because NB multiplies, one unseen feature would veto everything with a hard zero; Laplace smoothing (alpha=1 by default) guarantees no count is ever exactly zero.
What this session covers
Session 6 covers the naive Bayes material of the anchor sources - StatQuest's two NB episodes and sklearn's module 1.9 - with the theorem itself recapped from the statistics course. Videos, labs, and certificates stay with the official providers.