The launch is the middle, not the end
Most organisations treat a model launch like a finish line - the demo works, the deck lands, everyone moves on. That is exactly when the risk begins. A model learns from the past, and the past is always going away: Lumen adds a product line, a season turns, a pandemic upends how people shop, and the pattern the model learned last year no longer holds. If no one is watching, the model degrades silently and keeps making confident, wrong calls. A leader's job is to govern the whole life of the model - and this final session gives you the checklist to do it.
The lifecycle after launch 9 min live
A model is not a project with an end date; it is a loop that keeps turning. You frame the question, get and split the data, train, evaluate, and ship - and then the two steps everyone forgets: monitor the live performance, and retrain when it slips. The arrow runs back to the start, because the world never stops moving.
LiveDrift: why a good model goes bad3 min▶
A model learns the world as it was during training. When the world changes, the model does not - it keeps applying an expired pattern, confidently. That slow decay is called drift, and it is not a bug; it is the default fate of every deployed model.
- The inputs shift. Lumen launches a new product line or a new channel; the mix of visitors no longer looks like the training data.
- The relationships shift. A season turns, a competitor changes pricing, a pandemic rewrites shopping behaviour - and "returning customers buy more" stops being as true as it was.
- The model cannot tell. It will report the same confidence while quietly getting more wrong, because it has no idea the ground moved.
LiveMonitoring is not optional3 min▶
Because drift is inevitable, monitoring is not a nice-to-have - it is the price of running a model in production. You watch the live performance the same way you watch any other business metric, and you retrain when it slips past a line you set in advance.
- Watch the outcome. Track the model's real-world hit rate over time - for Lumen, weekly conversion among the sessions it flagged - not just the score it got at launch.
- Watch the inputs. If the incoming data starts looking different from the training data, that is an early warning the model is drifting.
- Set the trigger in advance. Decide the threshold at which you retrain before you are in the middle of a fire, and name who owns pulling that trigger.
Retraining is simply feeding the model recent data so it relearns the current world. It is routine maintenance, not failure - and skipping it is how a launch-day success becomes next quarter's liability.
Fairness, buy-vs-build, and governing the work 9 min live
A model inherits the world it was trained on, including its biases. It can quietly disadvantage a group without anyone intending it to. Governing ML means facing that squarely, choosing well between buying a model and building one, and asking the data team the handful of questions that surface risk before it reaches a customer.
LiveFairness and bias in one page3 min▶
A model has no values of its own. It faithfully reproduces whatever is in its training data - including patterns of disadvantage no one wrote down and no one intended.
- Bias enters through the data. If the past treated a group unfairly, a model trained on that past will learn to treat them unfairly too, and call it a pattern.
- Protected groups need explicit checks. A model can perform well on average and badly for a specific group. Average accuracy hides it; you have to look per group.
- Proxies are sneaky. Even without a sensitive attribute in the data, a model can reconstruct it from correlated features - a postcode standing in for something it should never use.
A company built a model to target its best hiring or marketing prospects and trained it on years of historical outcomes. Because those historical outcomes already favoured one group, the model quietly learned to score that group higher and steer opportunity away from everyone else - no one intended it, and the average metrics looked fine. Only a per-group check exposed the harm baked into the data.
LiveThe five questions to ask your data team3 min▶
You do not need to read code to govern a model. You need five questions, asked every time, before anything ships:
- 1 · What data did it learn from? And who or what might be missing or misrepresented in it?
- 2 · What is the baseline? How does it compare to doing nothing, or to a simple rule? A score alone means nothing.
- 3 · How was it tested? On held-out data it never saw? Checked per group, not just on average?
- 4 · What could go wrong? Where is it weak, what must it never be used for, and what does a confident wrong answer cost?
- 5 · Who is accountable? Who owns it, watches it, and decides when to retrain or retire it?
On buy versus build: a bought vendor model is faster and cheaper to start, but you inherit its data, its biases, and its blind spots - and you often cannot see inside it. Building your own costs more and takes longer, but you own the data, the card, and the accountability. The five questions apply either way; a vendor who cannot answer them is a red flag.
Read a Lumen model card ◐ demo
Here is everything the leader track taught, on one page. Below is a model card for Lumen's conversion model - the artefact a responsible team hands you before shipping. Read it the way the accountable exec should: not admiring the score, but interrogating the baseline, the limits, and the owner. Walk each row and ask "does this survive the five questions?"
| Row | What the card says | How an exec reads it |
|---|---|---|
| Task | Predict whether a checkout session will convert (buy). A classifier, scored per session. | A prediction problem, correctly framed. It predicts who is likely to buy - it does not tell us what would make them buy. |
| Features | Prior 30-day spend, new-vs-returning, channel, product category, device, pages viewed, session hour. | Prior spend and returning status carry the signal; session hour is near-useless noise. All are knowable at prediction time - no leakage. |
| Metric + baseline | ROC-AUC ~ 0.71. Beats the 96.8% "always predict no" baseline on recall, not on accuracy. | The honest comparison. Accuracy would have flattered a useless model; recall against the base rate shows it actually finds buyers. |
| Known limits | A real but modest signal - it ranks, it does not certify. Predicts conversion; does not prove any feature causes it. | Sets expectations. Use it to prioritise outreach, never to claim a lever. Causal questions go to an experiment. |
| Monitoring | Watch weekly conversion among flagged sessions and input drift; retrain when performance slips past the set line. | The lifecycle is owned, not assumed. There is a trigger and someone watching for drift, so the model will not rot in silence. |
| Owner | A named person on the data team, accountable for performance, fairness checks, and retrain decisions. | The most important row. A model with an owner has accountability; one without has none. |
This week ◐ 30 min total
- Audit a real one. Pick one ML use in your org - live or proposed - and run it through the five questions: what data, what baseline, how tested, what could go wrong, who is accountable.
- Fill in a model card. Using the six-row template above, write the card for that same use. The gaps you cannot fill are exactly where the risk lives.
- Find the unwatched model. Ask which live models in your org are being monitored for drift, and who owns them. If the answer is "no one", you found your first governance win.
- Optional: decide one buy-versus-build question your team faces, and defend it with the five questions rather than gut feel.
Three questions before you go 🎯 ◐ 90 seconds
1 · A conversion model worked well at launch but is quietly getting worse months later. What is going on?
This is drift. A model learns the world as it was; when the world changes (new products, seasons, a pandemic), the old pattern degrades. That is why monitoring and retraining are part of the lifecycle, not optional extras.
2 · How can a model end up treating a protected group unfairly even when no one intended it?
A model faithfully reproduces the patterns in its data. If the historical data encoded disadvantage, the model learns it and calls it a pattern - which is why per-group fairness checks matter, not just average accuracy.
3 · Before shipping a model, what is the most important thing a leader should establish?
The five questions - especially how it was tested, against what baseline, and what could go wrong - are what tell you whether a model is trustworthy. A high accuracy number alone (remember the 96.8% paradox) proves nothing.
What this session covers
This capstone teaches the framing behind the "ML in production and responsible AI" week of the major intro courses - the lifecycle, drift, fairness, and governance. The hands-on monitoring and MLOps tooling is deliberately beyond an intro; the sources below go deeper.