Why this session exists
Someone on your team will soon say "the model predicts...". This session makes sure you know what that sentence means, what it hides, and which decisions it quietly hands back to you. We cover the classic prediction toolbox - forecasting numbers and classifying labels - at executive intuition level: models as named tools with when-they-win, zero math, zero code. One honest boundary: deep learning, recommendations, search and GenAI are out of scope by design - they get their own AI literacy course later.
What supervised learning actually is 5 min live
Supervised learning is learning from labeled history to predict the next case. Show a model thousands of past customers WITH the answer attached (churned or stayed, and what they bought before), and it learns the pattern well enough to score the customers whose answer you do not know yet. Everything in this session is a variation on that one move.
LiveNumbers, labels, and the exam the model has not seen3 min▶
Two kinds of question, two kinds of model - that is the whole taxonomy you need today:
- Predict a NUMBER - regression. "What will next month's sales be?" "How many units will store 44 need?" Part 2 is this.
- Predict a LABEL - classification. "Will this customer churn - yes or no?" "Will this delivery be late?" Part 3 is this.
- Train/test split - the honesty mechanism. The team trains the model on, say, 80% of history and grades it on the 20% it never saw. That held-out chunk is the exam. A model graded only on data it trained on is a student grading their own homework.
- Overfitting - memorizing the textbook and failing the exam. The wiggly line above chases every random bump in history; it looks brilliant in the meeting and falls apart in production. The tell: great training score, mediocre test score.
Self-studyThe data is the model2 min read▶
The dirty secret of applied ML: the algorithm choice matters far less than the labels you feed it. Garbage labels in, garbage predictions out - forever, at scale, with confidence.
- The label IS a business decision. "Churn" is not a fact of nature. Is a customer churned after 30 days of silence? 90? After their contract lapses? Each definition produces a different model, a different call list, and a different budget. The data scientist will pick one if you do not - and theirs will be the convenient one, not the profitable one.
- Leaky labels flatter. If "cancelled auto-renew" sneaks into the training features, the model "predicts" churn brilliantly - by reading the answer off the exam sheet. Impossible-looking accuracy deserves suspicion, not applause.
- Old labels drift. A churn model trained on pre-price-change customers quietly rots after the price change. Ask when it was last retrained.
The exec takeaway: spend your meeting time on the label definition, not the algorithm name. That is the part you are qualified to own - and the part that moves the money.
Forecasting: predicting numbers 5 min live
Every forecast your team will ever show you is the same sandwich: a base level, a trend (drifting up or down), a seasonality (the repeating rhythm of weeks and holidays), and noise - the wiggle you accept instead of chasing. Decompose any forecast into those four and you can interrogate it like a native.
LiveLevel + trend + seasonality, and the humble baseline3 min▶
- The humble baseline: a moving average. "Next month = average of the last three months." Embarrassingly simple, surprisingly hard to beat. Any fancy model must out-predict this or it is theater. Always ask what the baseline was.
- Trend - the slow drift. Growing 2% a month? The forecast should lean up. Regression is the tool that finds this line.
- Seasonality - the repeating shape. November spikes, January slumps, Mondays differ from Saturdays. Miss it and every peak "surprises" you annually.
- Name-drop tier: ARIMA and Prophet. Your team's time-series tools - ARIMA is the classic statistician's engine, Prophet is Meta's business-friendly one. You will never touch either. Your job is to judge the output, not the math.
- MAPE - the honest scorecard. Mean absolute percentage error: on average, by what percent did the forecast miss? "Our demand forecast runs at 8% MAPE" is a sentence you can now use, benchmark, and put in a vendor contract.
Walmart forecasts demand store by store, item by item - feeding weather, holidays and local events (a home game near store 44 moves beer and pizza) into the models. The stakes are why they bother: a forecast that runs high becomes landfill and markdowns; one that runs low becomes empty shelves and customers who learned to shop elsewhere. Forecast error is not an abstraction - it is either waste or walkouts, and MAPE is how you watch it.
Classification: predicting labels 6 min live
Churn prediction is the canonical business classification problem - born in telecom (where contracts end monthly and switching is one phone call) and now standard kit in every SaaS and subscription business. The model reads each customer's history and outputs a score: how likely is this one to leave? What you do with the score is Part 3's real lesson.
LiveThe model ladder: explainability for accuracy3 min▶
Four names cover most of the classification your company will ever buy or build, arranged on one trade-off:
- Logistic regression - gives each factor a weight ("month-to-month contract adds risk, tenure subtracts it"). The auditable one; regulators and lawyers love it. When explainability is the requirement, it wins.
- Decision tree - readable if-then rules ("IF tenure under 6 months AND support tickets over 3, THEN high risk"). A manager can read it and argue with it - which is a feature.
- Random forest and gradient boosting - ensembles: hundreds of weak trees that vote. XGBoost and LightGBM are the brand names you will hear. Usually the most accurate, least explainable - the model equivalent of "trust the committee".
When someone proposes moving up the ladder, the question is not "is it more accurate?" (probably yes) but "what does the extra accuracy earn, and what does the lost explainability cost?" In a regulated decision, the answer often walks you back down.
LiveThe accuracy trap, and the two numbers that fix it3 min▶
Here is the trap that catches half of all ML buyers: with 5% monthly churn, a "model" that predicts "nobody churns" scores 95% accurate - and is perfectly useless. Accuracy lies whenever the thing you care about is rare. Hence the two honest numbers:
- Precision - of the customers we flagged, how many really were churners? Low precision = a call list full of false alarms = wasted save offers.
- Recall - of the customers who really churned, how many did we catch? Low recall = churners walking out uncontacted.
You cannot max both. Flag more people and recall rises while precision falls; flag fewer and the reverse. The dial between them is the threshold - and you will turn it yourself in Apply-along 2.
Telco churn programs run on exactly this dial. The save desk can call maybe 500 customers a week, each call costing agent time plus a retention discount. Set the threshold low and agents burn the week soothing customers who were never leaving; set it high and real churners slip past. The winning telcos treat the call list as a precision/recall budget decision reviewed monthly - not a model decision made once by the data team.
Fit the forecast yourself ★ 9 min · everyone fits
A year of monthly sales for one product line: holiday peak in Nov-Dec, winter dip after. Your job: beat the gut-feel forecast by tuning two sliders.
Look at the actual line first. Say out loud what you see: how steep is the drift, and where does the seasonal bump sit?
Slide trend only. Watch the MAPE readout move - a straight line can only do so much against a seasonal shape.
Now add seasonality. Watch MAPE drop as the forecast learns the Nov-Dec rhythm.
Goal: get MAPE under 6%. When you do, stop - resist the urge to chase the last wiggles.
Set the churn threshold yourself ★ 9 min · everyone decides
The model has scored 20 customers from most to least at-risk. Red dots will actually churn (in this exercise you get to peek at the future). Where you draw the line decides who gets the save call.
Slide the threshold LOW so nearly everyone gets flagged. You catch every churner - now count the save-offers wasted on customers who were staying anyway.
Slide it HIGH for a short, cheap call list. Precision looks great - now count the red dots to the right of your line: churners who walk uncontacted.
Find the threshold YOU would actually run, given that each save call costs a 20% discount. Defend it to your neighbor in one sentence.
Notice nobody in the room picked the same spot - and nobody is wrong. The right answer depends on margins and capacity, not on the model.
Spec one prediction ★ 4 min · everyone specs
Pick ONE prediction your business would pay for: demand next month? churn next quarter? which deliveries run late?
Write the label definition precisely - what exactly counts as "churned" or "late", measured when? (Part 1's self-study card is your checklist.)
Name the history the model would learn from: which tables, how many months, and whether the labels in them are trustworthy.
Price the misses in both directions: what does a false alarm cost you, and what does a missed case cost? Those two numbers ARE your future threshold conversation.
This week ◐ 25 min total
- Finish your prediction spec from Apply-along 3 with real numbers: label definition, data sources, cost of each miss direction.
- Ask your team one question about any model already running: "what are its precision and recall, and who chose the threshold?" Report the answer (or the silence) next session.
- Replay the forecast widget once - can you get under 6% MAPE in fewer moves, and can you say which slider earned each drop?
- Read the self-study card above: the data is the model - the churn-definition trap matters more than any algorithm choice you will ever face.
- Optional deep end: Provost & Fawcett, Data Science for Business, chapters 4-5 - overfitting and evaluation, told for decision makers.
Three questions before you go 🎯 ◐ 90 seconds
1 · Your data scientist says the new churn model "overfits". What is she telling you?
Overfitting is the wiggly line: perfect on history, useless on the future. The tell is a great training score paired with a mediocre score on held-out test data - always ask for the exam result, not the homework.
2 · A vendor's churn model is "95% accurate" and your churn rate is 5%. Why is the number worthless on its own?
Accuracy lies whenever the target is rare. Precision (of those flagged, how many real?) and recall (of the real ones, how many caught?) are the two numbers that cannot hide behind a rare event.
3 · Who should set the threshold that decides which scored customers get the save call - and on what basis?
The model produces scores; the threshold converts scores into spending. It is a budget decision about precision versus recall, and it belongs in a business meeting - revisited as costs and capacity change.
Concepts covered & their origins
This session teaches the working 80% of each idea at executive intuition level - the math stays with your data team, the judgment stays with you.