The scientist's questions, and the bench

Open. About the past or the present, answerable by anyone who runs the business. "What exactly runs out? Which products, which outlets, and how do you find out it happened?" It costs three to five minutes, it uncovers the facts behind its door, and it earns a point of trust, which is the currency the sensitive facts are priced in. Every open question in the bank carries a teach-back line.

Closed. Yes or no. "Is the van run weekly?" One or two minutes, one fact at most, and only the fact you already had the shape of. Cheap, and useful for confirming, but a meeting made of them learns nothing you did not walk in expecting.

Vague. "What would you like AI to do for you?" It sounds open. It is not, because it hands the design problem back to the person who came to you with it, and what comes back is a generic. Three minutes for a sentence you could have written yourself, and on the bench it reaches no fact at all.

Jargon. "What is the class imbalance on the positive label?" She has to ask what you mean, so it costs the question plus an explanation, five minutes rather than three, and she is a little less sure you understand her outlets. On the bench that is a point of trust lost, and it is the only kind that loses one. The irony is that it is a good question: it is the base rate, asked in a language she does not speak.

Leading. "So you want to forecast sales, right?" She says "I suppose so", because the question told her the expected answer and she is being polite. Nothing true came back. Something false went into your notes, and it costs later, when you train a model on the wrong target. On the bench a leading question records a belief with a cost attached, and the fast interview records five of them.

A scientist's output is a prediction somebody acts on. The facts that decide whether it is worth building are the exact target, how often it happens today, and what each wrong guess costs, and they live behind door 1. On Adeline's sheet the single most valuable fact, worth 5 of 68, is that the thing going wrong is a stock-out of one of about 40 chronic-medication products at outlet level, not sales. The next two are that it happens about 30 times a month across 38 outlets, found only as complaint calls, and that a missed stock-out and an over-order cost different amounts: a patient lost and a complaint on one side, expiry write-off she is measured on the other.

The three open questions that open that door, each with its teach-back line, word for word from the bank:

• "What exactly runs out? Which products, which outlets, and how do you find out it happened?" Teach-back: "I need the exact thing that goes wrong before I can promise to predict it. Sales and stock-outs are different questions with different data."
• "Roughly how often does it happen in a month, across all the outlets?" Teach-back: "How often it happens is the first number any prediction is judged against. If it is rare, a model that says 'never' looks accurate and is useless."
• "When it happens, what does it cost you? And what does it cost when an outlet orders too much?" Teach-back: "A wrong guess in each direction costs something different, and the model has to be built to the ratio between them, not to a single accuracy number."

Asked first, the first and third put 12 points in your notes by minute 9 and take trust to 2. The first alone returns the target and the base rate together, because she is already talking about complaint calls. The door is widest for this role because every modelling decision later, the label, the threshold, the evaluation, is a guess until these three answers exist.

Thirty-seven questions: twenty open (two to four per door), four vague, five jargon, five leading, three closed. The open ones are the sheet you built in session 1 with Adeline's answers already attached. The others are there because they are what gets asked in real meetings, and the bench needs them to show what they cost. The five jargon questions are all real modelling questions: labelled ground truth, class imbalance, a feature store, precision or recall, a rolling backtest. Every one is printed with its kind, its minute cost and the facts it reaches in the bank file the page loads; the widget below draws its buttons from that file, so what you click is what it counts.

Six Doors in printed order gets 46 of 68 with no false beliefs and every door opened. Tuning the order outcome first gets 55, a gain of 9, the largest role gain of the four sibling courses. The two orders open with the same question and part at the second. The printed order asks how often it happens, which returns only 2 new points because the first answer already mentioned the complaint calls. The tuned order asks what each wrong guess costs, and gets the fact that a missed stock-out and an over-order are not the same size.

In the printed list the cost question sits thirteenth, and the 45 minutes run out after the eleventh question is asked. So does her own test of success: the question "how would you know it worked?" costs 4 minutes and the printed order reaches it with 3 left. The tuned order asks it at minute 31 and hears that she counts the complaint calls herself and does not care what the accuracy number is. A printed order spends its first minutes well. It simply reaches "what does a wrong guess cost" too late, and for a scientist that is the question the whole model is built to.

Two hundred shuffles of the same bank average 27.5 and range from 0 to 46. The average is the meeting most people run when they have a good list and no order: some of the jargon lands early and costs trust, a leading question or two gets recorded, and a door stays shut. The best draw of two hundred only equals the printed order; none reaches the tuned one. The spread is the argument for an order at all. A method that scores 46 every time is worth more than one that scores 46 on its best day and 0 on its worst, and the bad days are the ones with the board in six weeks.

The ceiling, 61, is printed so the framework can be read honestly. It is an order found by a search that is allowed to read her fact sheet, which nobody in a real meeting can. Optimal against a known answer is not the same thing as good; the tuned framework is the thing you can do on Tuesday, blind, and get 55.

Five modelling questions in a row spend 25 of the 45 minutes and take trust to -3. The obvious cost is the minutes. The one that matters is what happens next. The first jargon question, "Do you have labelled ground truth for stock-out events?", is the right question: there is no stock history at all, because the spreadsheet is overwritten every Friday. She cannot answer it in that language. When the plain version comes later, "Is there any record of past stock levels, or only of sales?", trust is still too low and she does not volunteer it. The technical opener ends with 4 facts, 3 doors, and two sensitive facts unsaid: the missing stock history, and that the one IT person with the POS login is leaving in January.

The technical opener is not wrong because the questions are bad. Every one of them needs an answer before a model ships. It is wrong because they were asked of the wrong person, first, and the most important of them, whether the target was ever recorded, is the one that got lost.

A closed question that gets a "no" has taught you something. A leading question almost never gets a "no", because it carries its expected answer and the person across the table is being courteous, busy, or both. Rob Fitzpatrick's rules for customer conversations, as reported, apply without change: ask about specifics in the past, not opinions about the future, and treat agreement as the weakest kind of data. What comes back from a leading question is not information about her outlets; it is information about how agreeable she is, filed as if it were the first thing.

The bench prices each false belief at what building on it wastes. "The target is sales" costs 4: a model for a question nobody asked, and the second one Harbourline has paid for. "Daily predictions" costs 3: the van runs on Thursday, and a daily score changes nothing between runs. "Stock history is in the POS" costs 4: there is no stock history at all, and you find out when the training set will not join. "Optimise for accuracy" costs 3, because the two errors cost different amounts and she measures neither by accuracy. "The model replaces the pharmacist's check" costs 2, because the pharmacist looking at a shelf is the baseline any model has to beat, not the thing it removes. Five leading questions in six minutes, and the meeting stands at minus 16 before the first true fact.

Closed questions are not the villain; leading ones are. "Is the van run weekly?" is closed and gets a real fact for one minute, and the bench scores it so. The rule is about sequence: confirm with a closed question after an open one has shown you the shape of the answer, never before. And the anti-lever carries its own limit: a fast interview that asked only closed questions, no leading ones, would score higher than 15 and lower than the framework, because it would still learn nothing it did not walk in expecting. Note also that the fast interview opens all six doors. A door count on its own would call it a good meeting, which is why the counters are read net value first. The bench does not pretend that speed is the sin. Assuming is.