Where we start
Session 1 put the first deployment in the middle of the project. This session is about the left edge, the only stretch of the timeline where saying no costs a meeting instead of a quarter. Three questions decide it. Is there a real problem with an owner, data and labels behind it? What is the simplest rule that already solves part of it, and how well does that rule score? And what number, agreed in advance, would make a model worth its running cost? A project that cannot answer those three is not ready for a model, however good the demo.
★ What you walk out with tonight. A one-page scoping memo for the project you brought: the problem in one sentence with no AI words, the cost of a wrong yes and a wrong no, the no-model baseline measured at your real label delay, the business number the model has to move, and a stop rule with a date on it. Plus the habit that saves the most money in ML: measure the baselines before anyone trains anything.
Scoping is where no is still cheap
5 min liveThe decision first: before a project gets a team, a leader should see it pass two checks, feasibility and value, and see the date at which it will be stopped if it does not deliver. Andrew Ng's Machine Learning in Production course puts scoping in week 3 as an optional set of five videos: "What is scoping?", "Scoping process", "Diligence on feasibility and value", "Diligence on value" and "Milestones and resourcing". The titles are read from the course page; the lecture content is reported, from compiled notes, since the official notes sit behind the course login. The titles alone make the point: value gets its own video.
LiveBrainstorm problems before solutions2 min▶
Write the problem the way the business owner would say it, and ban the words AI, ML, model and predict from that sentence. "Reduce the money we lose when we buy electricity forward at the wrong price" is a problem. "Use ML to forecast prices" is a solution looking for one. The ban is practical: once the sentence names a technique, every alternative that does not use it has been ruled out before anyone compared them.
Then list solutions, plural. A rule, a process change, a better report, a human review queue, a model. The model is one row in that list, and step 3 and step 4 are how it earns its place.
LiveFeasibility: is there anything to learn from?2 min▶
Four questions, each with a written answer rather than a nod:
| Question | Why it can stop the project |
|---|---|
| Is there data? | The inputs the model would use must exist at prediction time, not only in a warehouse a week later |
| Are there labels? | Someone or something has to record what actually happened, consistently, for enough cases |
| What is the label delay? | How long after a prediction you learn whether it was right. It sets the baseline and the retrain policy; session 4 measures how much it matters |
| Is there a baseline, or human-level performance? | If nobody can say how well the job is done today, nobody can say whether the model improved it |
LiveValue: bridge the technical metric to a business number2 min▶
Technical teams measure accuracy, precision, error. The business measures money, time, risk, customers. Value diligence is writing the bridge between them, "every accuracy point above the baseline is worth this much, because...", and then naming the number that would make the project worth shipping and running. That number is agreed with the business owner before modeling starts, not negotiated after the results are in.
The decision you sign: the project gets a team only when the value number has a signature next to it. If nobody on the business side will put their name to a number, that is the finding.
Self-studyMilestones and resourcing, with a stop date3 min▶
Milestones turn a scoped project into dated checkpoints: data access confirmed, baseline measured, first model compared against it, a decision to deploy or not. Resourcing names the people and the running cost, which session 1 argued is mostly after launch.
The milestone that matters most is the one teams least like writing: the date at which the project stops if it has not beaten the baseline by an agreed margin. Written at scoping, it is a plan. Written after three months of work, it is a fight, and the sunk cost usually wins it.
A 2 percent gain found by fixing data, not by modeling. In Zinkevich's Rules of Machine Learning, Rule 10, "Watch for silent failures", describes a table behind Google Play that went stale for 6 months. Nothing crashed. Refreshing the table alone gave "a boost of 2% in install rate". No new model, no new feature, no research. It is the reason feasibility asks about the data before the model: the value was sitting in a pipeline nobody was watching.
Every model has a rule to beat
5 min liveThe decision first: no model result reaches a leader without the baseline printed next to it, measured on the same rows at the same label delay. A baseline is the score you get without the model. It is the only honest reference point, and on real data it is often uncomfortably high. Here is the session 4 bench, a replay of a real electricity market, with the no-model rule at two label delays.
LiveFour kinds of baseline, and which to demand2 min▶
| Baseline | What it is | On the bench |
|---|---|---|
| Majority class | Always predict the most common outcome | Always DOWN, 57.72 |
| A rule a clerk could run | A heuristic with no training: repeat the last known answer, a threshold, a lookup | Repeat the last known label, 65.35 at a 1-day delay |
| Human-level performance | How well the person doing the job today does it, measured on the same cases | Not measured on Elec2; for your project, go and measure it |
| The system in place today | Whatever the business actually runs now, however unglamorous | The number the model has to beat to change anything |
Demand at least the first two on every project, plus human-level whenever a human does the job today. The model's result means nothing until it sits in that table.
LiveThe Elec2 lesson: the baseline depends on the label delay3 min▶
Elec2 is one of the most used datasets for testing models that adapt to a changing world: 45,312 half-hours of the New South Wales electricity market, labelled UP or DOWN. In 2013 Indrė Žliobaitė checked it against the simplest possible rule, "repeat the current label", which she calls a moving average of one. It scores 85.3 percent on the full file, against 51 percent if the labels were independent. The file had been used in over 40 concept drift experiments, and she recommended that every result on it be compared against that rule.
The bench's thinned rows, one every 1.5 hours, tell the same story with a twist. The same rule scores 74.63 when the label arrives 1.5 hours later, and 65.35 when it arrives a day later. Same data, same rule, nine points apart. The baseline is not a property of the data; it is a property of the data and how fast you learn the truth.
The decision you sign: a baseline measured at a label delay the business does not have is not a baseline. Ask which delay it was measured at.
Self-study"A simpler model is not better" is a test, not a hope2 min▶
Breck and colleagues' ML Test Score, from session 1, lists it as model development test 5: "A simpler model is not better". It is written as something a team runs and records before launch, not a belief it holds. Train the simple thing, score it on the same rows, and keep the evidence. Paired with a baseline table, it turns "the model is good" into "the model beats these specific alternatives by this much", which is the only sentence a launch review should accept.
A hundred and thirty retrains, beaten by a copy. On the session 4 bench, the weekly-refit model retrains 130 times over two and a half years and scores 73.55 at a 1-day label delay. If the same market's labels arrived within 1.5 hours, repeating the last label would score 74.63 with no model at all. A team that measured only at its own convenient delay could ship the model and never learn that a data feed was worth more than the training pipeline.
Launch without ML, then earn it
5 min liveThe decision first: the default launch is the simplest thing that works, with the metrics wired in, and machine learning is an upgrade the project earns with data. Martin Zinkevich's Rules of Machine Learning, 43 rules from Google engineering practice, open with exactly this order. Read as a stack, each rule is a layer that only makes sense on top of the one below it.
LiveRule 1: "Don't be afraid to launch a product without machine learning."2 min▶
The rule's own summary is blunt: "If machine learning is not absolutely required for your product, don't use it until you have data." The reason is the data, not modesty. A heuristic in production collects the inputs and the outcomes a model will later need, and it tells you what the baseline really is.
The same rule carries the line that gets quoted most: "If you think that machine learning will give you a 100% boost, then a heuristic will get you 50% of the way there." Say plainly what it is: an illustrative rule of thumb from an experienced engineer, not a measurement. No study sits behind the 50 percent. Use it to set expectations, never as a forecast in a business case.
LiveRules 2 and 3: metrics first, and the trigger for ML2 min▶
Rule 2 is "First, design and implement metrics." Before the first model, the system should already be recording the numbers the model will be judged on, so the heuristic's score is known and the model's gain can be read against it rather than guessed.
Rule 3, "Choose machine learning over a complex heuristic", names the moment to upgrade: "A simple heuristic can get your product out the door. A complex heuristic is unmaintainable." When the rule list grows into special cases nobody can reason about, a model is the cheaper thing to maintain.
The decision you sign: the first launch uses the simplest rule that works, with the metrics live. The model is funded when the heuristic has become complex, or the data says it is leaving value behind.
Self-studyRules 4 and 16: keep it simple, then keep shipping2 min▶
Rule 4, "Keep the first model simple and get the infrastructure right", explains why: "The first model provides the biggest boost to your product, so it doesn't need to be fancy." Most of the early risk is in the pipeline, not the algorithm, which is session 1's small box again.
Rule 16, "Plan to launch and iterate", sets the cadence expectation: "Many teams have launched a model per quarter or more for years." A scoping memo that plans one launch and no second one has planned the easy half.
When the model does earn it. Rule 1 is not a verdict against ML. On the session 4 bench at a 1-day label delay, the weekly-refit model scores 73.55 against 65.35 for repeating the last known label: 8.20 points the rule cannot reach. That is what earning it looks like, a measured gap over the no-model rule at the delay the business really has, not a demo.
The one-page scoping memo
★ 22 min · everyone buildsTake the project you brought, the one that has not launched yet. If you have none, use the worked example after the steps. Six blocks, each copied into a document and filled in. The memo fits on one page on purpose: if a block needs a second page, the project is not scoped yet. Block 6, the stop rule, is the one to write before you leave the room.
Write the business problem in one sentence, with no AI words. No "AI", "ML", "model" or "predict". Then name the person who owns the problem, the one whose number moves if it is solved.
Name the decision the model would make, and what each kind of mistake costs. A wrong yes and a wrong no rarely cost the same. If nobody can price either, the value block in step 5 will not survive contact with the business.
Measure the baselines. The no-model rule first, measured at the label delay you really have, not the one that flatters it. Add human-level performance if a person does the job today.
Check feasibility. Give the label delay as a median and a 90th percentile: the slow tail is what decides the retrain policy in session 4.
Bridge the technical metric to the business metric. Write the chain in words, then the number that would make the project worth shipping and running. Get the owner from block 1 to sign it.
Set milestones, resourcing and the stop rule. Dates and names for each milestone, then the sentence that ends the project if it does not deliver. Written now, stopping is a plan; written later, it is a fight.
Northline Grid, a constructed electricity retailer. Constructed company, real bench numbers. Northline buys electricity forward and wants to know whether the New South Wales price will go UP or DOWN against its 24-hour average, to decide when to hedge. Block 1: "we lose money when we buy forward just before the price falls", owned by the head of trading. Block 3: the no-model rule, repeat the last known direction, scores 65.35 on the bench at a 1-day label delay, which is when Northline's settlement data arrives; if its data feed delivered labels within 1.5 hours, the same rule would score 74.63, above both bench models. Block 5 would carry Northline's own value number. Block 6, with a constructed margin: "if by the end of the quarter the model does not beat 65.35 by 3 points at a 1-day delay, we stop, and we price a faster data feed instead." The margin and dates are constructed; your numbers will differ.
Measure the baselines before anyone trains anything. On Elec2 it takes five lines, and the outputs below are real, measured on the full file.
The second line is the rule if labels arrive at once: copy the last half-hour, 85.33 percent, Žliobaitė's moving average of one. The third is the best you can do by copying if labels arrive a day late: copy the same half-hour yesterday, 65.62 percent. The model's target is whichever line matches your label delay. For your own data, shift by your real delay in rows. Once a model exists and beats the right line, the next step is looking at what it gets wrong, which is error analysis, taught in ML Strategy session a3 ↗.
Put the stop rule in the calendar. A stop rule in a memo gets forgotten; a stop rule with a meeting invite on its date gets read. Session 6 comes back to it, because the kill decision is much easier when it was agreed before anyone had a stake in the answer.
Before session 3
30 min- Write the memo for the project you brought, all six blocks, on one page. If a block will not fit, that block is the one to scope further.
- Measure the no-model rule on your own data at your real label delay. Shift by the delay you actually have, not by one row. Write both numbers down if the team has been quoting the fast one.
- Get the business owner to sign the value number. In person, on the memo. A number nobody will sign is a finding, not a formality.
- Write the stop rule and put a date on it, then put that date in the calendar of everyone who signed.
Three questions
Clear all three and session 2 gets stamped in your passport.
1 · A business case says "Google found a heuristic delivers 50% of ML's benefit, so our rule-based version will capture half the value." What is wrong with it?
The line illustrates why launching without ML is reasonable. No study sits behind the 50 percent; the business case needs its own baseline measured on its own data.
2 · On the session 4 bench, repeating the last known label scores 74.63 with labels 1.5 hours late and 65.35 with labels a day late. What does that tell a team scoping a model?
Same data, same rule, nine points apart. A baseline measured at a faster delay than you have makes the model look worse than it is; one measured slower makes it look better.
3 · Where does the stop rule "if by this date the model does not beat the baseline by this margin, we stop" belong?
Written at scoping, the stop rule is a plan. Written after months of work, the sunk cost argues against it every time.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 2 cheat sheet · pin this
A scoped model still has to meet real traffic. Session 3 decides how: the degrees of automation from human only to full automation, shadow mode, canary and blue-green rollouts, and the rollback written before the rollout. Bring the decision from block 2; its cost of a wrong yes decides how carefully it should go live.