learn-intro-ml-with-phoebe / Leader session 5 of 6
Learn Intro to Machine Learning with Phoebe · Leader session 5 of 6

Prediction is not proof, and simple usually wins

Two things every leader must hold at once. The first is a rough map of the model zoo - enough to know when a plain logistic regression is the right call and when a problem genuinely needs the heavy machinery. The second is the hardest line in all of machine learning: a model predicts, but it never proves what causes an outcome. Confuse the two and you will spend real budget chasing a correlation that moves nothing. This session gives you the map and draws the line - no code, just the judgement that keeps a good prediction from becoming a bad decision.

🟠 Getting real Leaders No code 45 min
0-3 · Setup 3-24 · The model zoo 24-42 · Predict vs cause 42-45 · Wrap
Part 0

The most expensive mistake in the room

A model tells Lumen Skincare that visitors who arrive from email convert far more often than average. The obvious move is to pour budget into email. It is also, often, the wrong move - because the model has spotted a pattern, not a lever. Email may simply be where Lumen's already-loyal customers happen to show up; shift the budget and you may get nothing. This session arms you against exactly that error, and gives you the map to pick the simplest model that could possibly work before anyone reaches for a neural network.

Live - covered in session Self-study - read after ★ The one big idea The case: Lumen Skincare
★ What you walk out with today A one-glance map of the model families and when each earns its complexity, the instinct to start simple and interpretable every time, and - most valuable of all - the discipline to separate "the model predicts X" from "X causes the outcome". You will know, in the meeting, when the honest next step is not a bigger model but an experiment.
Part 1 · the map

The model zoo at a glance 9 min live

You do not need to know how each model works. You need to place them on two axes: how complex they are to run and trust, and how easily a human can read why they decided what they did. That map tells you which family to reach for - and which ones are a whole course of their own.

simple → complex black-box → interpretable Linear / logistic simple, interpretable, the baseline Decision trees readable rules, overfit easily Ensembles random forests, boosting (XGBoost) - strong, opaque Neural nets / deep learning huge data, images/text, black-box Clustering / k-means unsupervised - find segments Move down-right only when a simpler model has been tried and genuinely falls short. Complexity costs interpretability, and you pay it in trust.
🔍 Click to zoom - the model families, mapped by complexity and interpretability
LiveStart simple3 min▶

The right first model is almost always the simplest one that could work - logistic regression for a yes/no, linear regression for a number. Not because it is the most powerful, but because it is honest about what it knows and cheap to reason about.

  • Linear / logistic regression - simple, fast, and interpretable. You can read the weights and see which features push the prediction. Always your baseline.
  • Decision trees - a flowchart of if/then rules a human can follow. Readable, but they overfit easily, memorising the training data unless kept shallow.

A simple model that everyone understands and that beats the baseline is worth more than a complex one no one can explain. Start here, prove there is signal, then earn the right to add complexity.

The leader's default When a team opens with "we'll use deep learning", ask "what did the logistic regression score?" If they skipped it, they skipped their own baseline - and they cannot tell you whether the fancy model is actually adding anything.
LiveWhen you actually need the heavy models3 min▶

The complex families exist for a reason - just not usually a first project's reason. Reach down-right on the map when a simple model has genuinely fallen short and the problem justifies the cost:

  • Ensembles (random forests, gradient boosting like XGBoost) - many trees combined. Often the strongest on tabular business data, at the cost of interpretability. Their own course: ensemble-methods.
  • Neural networks / deep learning - for huge datasets and unstructured inputs like images, audio and text. Powerful, data-hungry, and a black box. Their own course: deep-learning.
  • Clustering / k-means - unsupervised: no labels, it finds segments no one defined. A different job entirely, covered in unsupervised.

Naming these is enough for now. An intro course teaches the workflow and judgement; the heavy machinery is a deliberate next step, not a prerequisite.

Real world

Lumen's conversion problem is tabular checkout data with a modest signal. A logistic regression captures nearly all of it (ROC-AUC ~ 0.71); a deep neural network would add enormous complexity, demand far more data, and likely land in the same place - while nobody could explain a single decision to the marketing team. Simple wins, and it wins the trust too.

Part 2 · the hard line

Prediction is not causation 9 min live

Here is the line that separates a data-literate leader from a fooled one. A model exploits correlations to predict an outcome. It cannot tell you what would happen if you changed something. Predicting who will buy is not the same as knowing what would make them buy - and only one of those questions a model can answer.

What a model does: predict Features correlate Prediction Answers "who is likely to buy?" Cannot tell you why, or what to change. What an experiment does: prove Intervene two groups Causal lift Answers "what would change buying?" The only way to measure a true cause. A model reads the world as it is; an experiment changes the world and watches. To pull a lever, you need the right-hand panel.
🔍 Click to zoom - a model predicts the world; an experiment changes it
LiveImportance is not a lever3 min▶

Models hand you a "feature importance" list - which inputs mattered most to the prediction. It is genuinely useful, and it is genuinely dangerous, because it is not a list of causes.

  • Importance means "predictive", not "causal". "Returning customer" is Lumen's second-strongest feature. But labelling someone returning causes nothing - it is a marker of a customer who was already loyal, not a lever you can pull.
  • Correlation gets its predictive power for free. A feature can ride along with the outcome without moving it. The model does not know or care about the difference; it only wants to predict.
  • The trap: reading a high-importance feature as an instruction ("make more customers 'returning'") and spending budget on it.
The one-line check For any important feature, ask: "If I could change this for a customer tomorrow, would the outcome actually move - or is it just a sign of who they already are?" If it is a sign, it is not a lever.
LiveWhen you need an experiment, not a model3 min▶

The moment a question contains the word "cause" - or its disguises, "would", "if we", "does X drive Y" - you have left prediction and entered causation. No model can answer it. You need to intervene and measure.

  • "Which visitors will buy?" - a prediction. A model is exactly right.
  • "Would a discount cause more of them to buy?" - a causal question. Only an experiment - randomise who gets the discount, compare the groups - can answer it.

This is the explicit bridge to the next course. learn-experimentation-with-phoebe is entirely about answering the causal questions that machine learning cannot. Knowing which of the two you are holding is the leader's job.

Real world

Lumen's model showed that email visitors convert more, so leadership shifted a chunk of budget toward email. Conversion did not move. The reason: email was not causing purchases - it was simply where Lumen's already-loyal buyers arrived. The model had read a correlation perfectly and told them nothing about cause. Only an A/B test - randomly withholding email from some loyal customers - would have revealed that the lever was a mirage.

Before Session 6

This week ◐ 25 min total

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · For a first conversion model on Lumen's tabular checkout data, which should you reach for first?

Start simple. Logistic regression is fast, interpretable, and sets the baseline that tells you whether any fancier model is actually adding value. On modest tabular signal, it often captures nearly everything.

2 · "Returning customer" is a top feature in Lumen's model. What does that prove?

Feature importance is about prediction, not causation. "Returning" is a marker of an already-loyal customer, not a lever you can pull. Reading importance as a cause is the expensive mistake.

3 · "Would a discount CAUSE more visitors to buy?" Why can't a model answer this?

Anything asking what a change would cause needs an intervention - randomise who gets the discount and compare groups. A model reads the world as it is; only an experiment changes it and measures the lift. That is the next course.

Source material

What this session covers

This leader session teaches the framing behind the "which model, and what a model can't do" week of the major intro courses - the model families and the prediction-versus-causation line. The heavy families and the causal tests are whole sibling courses; the sources below map where to go next.

✓The model zoo mapped by complexity and interpretability; start simplePart 1
✓Prediction is not causation; importance is not a lever; when you need an experimentPart 2
◐The heavy families map to sibling courses: ensemble-methods, deep-learning, unsupervisednamed here, taught there
◐The causal question leads to learn-experimentation-with-phoebethe next course in the ds ladder

Leader Session 5 cheat sheet · pin this

The model zooLinear/logistic (simple, interpretable), trees (readable, overfit), ensembles like XGBoost (strong, opaque), neural nets (huge data, black-box), clustering (unsupervised segments).
Start simpleLogistic/linear regression first, always. It is the baseline that tells you whether anything fancier is actually earning its complexity.
Heavy models are next coursesEnsembles, deep learning and unsupervised each have their own course. An intro teaches the workflow, not the zoo.
Predict ≠ proveA model exploits correlation to predict. It never establishes what causes an outcome. Hold the two apart.
Importance is not a leverA high-importance feature ("returning") can be a marker of who someone is, not something you can change to move the result.
When you need an experimentAny "would / cause / does X drive Y" question needs a controlled test, not a model. That is learn-experimentation-with-phoebe.