The gap, from the scientist's chair

Newton's tappers were not bad at tapping. They could not un-know the song while they tapped, so they judged the listener's chances from inside their own experience. Camerer, Loewenstein and Weber had named this a year earlier: the curse of knowledge, the finding that better-informed people cannot fully set their information aside when predicting what less-informed people will understand.

A data scientist carries a very loud tune. "Predict" arrives with a whole melody attached: a target variable, a row per outlet per day, a probability, a threshold, a score on held-out data. Adeline's tune is just as loud and completely different. To her, "predict" means "stop getting caught by empty shelves": a patient on chronic medication turns up, the shelf is empty, the patient goes to a competitor and the complaint comes to her. She has no reason to say which product, how often, or what an over-order costs, because inside her tune those are obvious.

So the scientist fills the silence with his own tune, and the model gets built to it. The gap is not closed by teaching her the vocabulary. It is closed by questions that make her tune audible: what exactly runs out, how often, and what each kind of mistake costs, asked in words that need no translation, with the reason for asking said out loud.

Her request starts with the board: "The board keeps asking about AI." Behind that sentence, on her side of the table, sit two different projects. The board wants the word AI on a slide. She wants fewer calls. A scientist who hears only the first builds something impressive for a board meeting; one who hears only the second might solve it with a spreadsheet and leave her with nothing to show upward.

Neither is wrong, but they are different builds with different tests, and she will not separate them for you unless you ask. The professional move is to name both out loud early ("it sounds like there is a board question and an operations question; can we take the operations one first?") and let her choose which one the first project serves. Session 5 comes back to this when the three candidate projects are drawn, because one of them is usually not a model at all.

The first fact is the thing that goes wrong, named precisely enough that you could tell whether it happened yesterday. At Harbourline it is a stock-out of one of about 40 chronic-medication products at outlet level: a patient turns up and the shelf is empty. Not sales, not demand in general, not every product in the shop. That sentence already scopes the model to about 40 products across 38 outlets, and it rules out the vendor's forecast before you have seen a row of data.

The question that gets it is plain: "What exactly runs out? Which products, which outlets, and how do you find out it happened?" The last clause matters most. How she finds out is how the target is recorded, and at Harbourline the answer is that nobody records a stock-out anywhere. The only record is the complaint call. You have just learned that your label does not exist yet, in the words of a person who has never heard the word label.

The second fact is how often it happens today. Adeline's answer is roughly 30 a month across the 38 outlets, and she knows because each one becomes a complaint call to her. Do the arithmetic on the constructed sheet: about 40 products in 38 outlets is 1,520 outlet-product pairs, and 30 a month is about seven a week. In any given week, well over 99 percent of pairs do not run out. A model that answers "never" to every one of them is right more than 99 percent of the time and useless every time.

That is why the base rate is asked early, before anyone mentions accuracy. It is the first number any prediction is judged against, and on a rare event it tells you in advance that a single accuracy figure will flatter whatever you build. It also tells you what a real improvement looks like: fewer of her 30 calls, not a higher score.

Douglas Hubbard's definition of measurement is the reason this belongs at door 1 rather than in the model review: a measurement is a quantitatively expressed reduction of uncertainty based on one or more observations. The uncertainty here is hers, about which shelves will be empty next week. A prediction that does not reduce it has measured nothing, however well it scores.

The third fact is what each kind of wrong guess costs, and the two are never the same size. At Harbourline a missed stock-out costs a patient a trip to a competitor and costs Adeline a complaint to head office. An over-order costs expiry write-off on medication that sits too long, and head office measures her on write-off. A model that flags every outlet every week would stop the stock-outs and wreck the number she is judged by.

You do not need the cost matrix as a formal object in the first meeting, and she should never hear the word. You need the two bills in her words, and ideally which one she would rather pay more of. "When it happens, what does it cost you? And what does it cost when an outlet orders too much?" Treating the ratio between those costs as a design decision in its own right is the subject of Decision Intelligence ↗; here the job is only to get both numbers onto your sheet before anyone says "optimise for accuracy".

Google's machine learning problem-framing course gives two pieces of advice that belong in the first meeting, not the project plan: check whether a non-ML solution solves the problem before building a model, and define the success metrics before building. Both are outcome-door questions, and both come back with facts that change the project.

The non-ML solution at Harbourline already exists. Today the pharmacist looks at the shelf and WhatsApps the area manager, and that catches most of them. Any model has to beat a person looking at a shelf, so the current way is not a nuisance to be replaced; it is the baseline you are measured against, and it is door 3. The success metric exists too, in her words: the complaint calls about empty shelves go down, and she counts them herself. She does not care what the accuracy number is. Agreeing that test in the first meeting is what stops a later argument about precision and recall that she cannot follow and should not have to.

Sharpening a question once it has been handed to you, and recognising when no column in any system answers it, is taught in Data Thinking ↗. The skill here comes one step earlier: getting the question out of a person who did not know she was asking it.

A teach-back line is plain (no word she would have to ask about, so "how often" not "base rate", "a wrong guess" not "a false positive"), true (a real principle of the craft, not a pleasantry) and one sentence, or two short ones at most, because a paragraph is a lecture and she stops answering.

The three outcome questions in the bank each carry one, and they are worth learning word for word. Before "What exactly runs out?": "I need the exact thing that goes wrong before I can promise to predict it. Sales and stock-outs are different questions with different data." Before "Roughly how often does it happen in a month?": "How often it happens is the first number any prediction is judged against. If it is rare, a model that says 'never' looks accurate and is useless." Before "What does it cost you, and what does it cost when an outlet orders too much?": "A wrong guess in each direction costs something different, and the model has to be built to the ratio between them, not to a single accuracy number."

Between them, those three sentences teach her the target, the base rate and the cost ratio, the three ideas a scientist most needs a sponsor to understand, and none of them uses a word from a textbook. Session 4 builds more of these and shows how to pick the one concept per meeting worth explaining properly.

Session 3 sorts every question in the bank into five kinds. Open questions about the past or present uncover facts and earn trust. Closed yes-or-no questions are cheap and get one fact at most. Vague ones ("what would you like AI to do for you?") cost minutes and bring back a generic. Jargon costs twice: "what is the class imbalance on the positive label?" gets nothing, and she is a little less sure you understand pharmacies. Leading questions are the dangerous kind. "So you want to forecast sales, right?" gets a polite yes, and "the target is sales" goes into your notes and later into your build. The fast yes-or-no run on the bench leaves five false beliefs like that behind it.

The trust is not decoration. Six of the 24 facts on Adeline's sheet only come out once some trust exists, including how often stock-outs happen and that the stock spreadsheet is overwritten every Friday. The most sensitive one needs the most trust of all, and it decides whether anything you build survives the quarter; session 3 shows which it is and what it takes to hear it.