The AI developer's questions, and the bench

Open. About the past or the present, answerable by anyone who runs the shop. "Show me one thread from yesterday, start to finish." It costs three to five minutes, it uncovers the facts behind its door, and it earns a point of trust, which is the currency the sensitive facts are priced in. Every open question in the bank carries a teach-back line.

Closed. Yes or no. "Is it three area managers?" One or two minutes, one fact at most, and only the fact you already had the shape of. Cheap, and useful for confirming, but a meeting made of them learns nothing you did not walk in expecting.

Vague. "What would you like AI to do for you?" It sounds open. It is not, because it hands her your job, and what comes back is a generic: the board wants AI on a slide. Three minutes for a sentence you could have guessed.

Jargon. "What is your tolerance for hallucination rate?" She has to ask what you mean, so it costs the question plus an explanation, five minutes rather than three, and she is a little less sure you understand her business. On the bench that is a point of trust lost, and it is the only kind that loses one.

Leading. "So the assistant should reply to the suppliers automatically?" She says "I suppose so", because the question told her the expected answer and she is being polite. Nothing true came back. Something false went into your notes, and it costs later, when you build a bot that says yes on its own. On the bench a leading question records a belief with a cost attached, and the fast interview records five of them.

An AI developer's output replaces or assists a human step. The facts that decide whether it is safe are the thread as it happens today, which replies are lookups and which are decisions, how often the person catches an error, and who checks the output. On Adeline's sheet the single most valuable fact, worth 5 of 69, is one real thread end to end: the supplier sends a confirmation, the area manager checks it against the outlet's order in a different WhatsApp group, and replies OK or queries; about forty threads a day per manager, and about four in five end in a plain OK. Right behind it, and only once she trusts you a little: about one confirmation in twenty has a wrong price or a substitute in it. And from the decision door: only the area manager may accept a substitute or a price change, and a wrong substitution is a compliance problem, not a customer-service one.

The four open questions that open those facts are the first four in the AI-developer-tuned order on the bench, each with its teach-back line:

• "Show me one thread from yesterday, start to finish. What did you do at each message?" Teach-back: "The workflow as it happens today is the spec. I cannot automate a step I have not watched a person do."
• "How often is a confirmation wrong, and how do you catch it?" Teach-back: "The error rate today is the bar any assistant has to beat, and how you catch errors is the part the assistant has to keep."
• "In a thread, which replies need a person to decide, and which are just looking something up?" Teach-back: "I split every workflow into lookups and decisions. Assistants do lookups, people keep decisions, and the line between them is yours to draw."
• "What is the worst thing that happens if a reply is wrong?" Teach-back: "The worst case sets how much checking the assistant needs. A wrong delivery time and a wrong drug are not the same mistake."

Asked first, they put 23 points in your notes by minute 17 and take trust to 4, past the level the sheet's most sensitive fact is priced at.

Thirty-seven questions: twenty open (two to five per door, five of them behind the current way), four vague, five jargon, five leading, three closed. The open ones are the sheet you built in session 1 with Adeline's answers already attached. The others are there because they are what gets asked in real meetings, and the bench needs them to show what they cost. Every one is printed with its kind, its minute cost and the facts it reaches in the bank file the page loads; the widget below draws its buttons from that file, so what you click is what it counts.

Six Doors in printed order gets 54 of 69 with no false beliefs and every door opened. Tuning the order for an AI developer, current way first, gets 55, and 23 in your notes by minute 20 instead of 22. One point. It is small for a plain reason: Adeline's request is already about the current way, so the printed order gets there early anyway. By minute 12 it has already asked which replies are lookups and which are decisions, and by minute 16 the next question has brought back the one-in-twenty error rate that sits behind the current-way door. There is not much left for a better order to win.

The ceiling is 63, from a search that is allowed to read the fact sheet, and here it does something worth noticing: it opens with the lookups-versus-decisions question, which a real interviewer could ask first with no sheet at all. For this role the framework and the optimizer agree on the opener and differ on what follows; the extra eight points come from knowing which later question reaches which hidden fact, and nobody in a real meeting knows that. The framework is the thing you can do on Tuesday, blind, and get 54.

Two hundred shuffles of the same bank average 30.1 and range from -1 to 52. The average is the meeting most people run when they have a good list and no order: some of the jargon lands early and costs trust, a leading question or two gets recorded, and a door stays shut. The spread is the argument for an order at all. A method that scores 54 every time is worth more than one that scores 52 on a good day and -1 on a bad one, and for an assistant that will talk to 36 suppliers, the bad days are the ones that reach a shelf.

Put the numbers side by side: the technical opener 11, the efficient interview 21, the Six Doors 54. The order of the good questions moved the score by one point; the kind of question moved it by more than forty. That is the lesson of the bench for an AI developer.

The technical opener shows why. Five questions in a row about fine-tuning, hallucination tolerance, eval sets, vector stores and guardrails spend 25 of the 45 minutes and take trust to -3. The obvious cost is the minutes. The one that matters is what happens next: by the time an open question about the product list is asked, trust is still below zero, so when the question that could have surfaced Kenneth's January exit comes, she holds it back. The meeting ends with four facts and two doors, and the thread itself, the current way, was never opened. Every one of those five is a question you will need answered, most of them by yourself in week three. They were asked of the wrong person, first.

A closed question that gets a "no" has taught you something. A leading question almost never gets a "no", because it carries its expected answer and the person across the table is being courteous, busy, or both. Fitzpatrick's rule for customer conversations applies without change: people will say yes to be polite. What comes back is not information about her business; it is information about how agreeable she is, filed as if it were the first thing.

The bench prices each false belief at what building on it wastes. "The assistant replies to suppliers on its own" costs 4: a bot that says yes to a substitute nobody checked, when only the area manager may accept one and each needs a record by law. "The orders are in English" costs 3: a matcher that cannot read Malay, Chinese or the outlets' product nicknames. "There is an order table to match against" costs 4: there is none, only another WhatsApp group. "Mistakes are rare and cheap" costs 3: about one in twenty is wrong, and a wrong drug is not cheap. "The managers will use a separate app" costs 2: they live in WhatsApp, in a van. Five leading questions in six minutes, and the meeting stands at minus 16 before the first true fact.

Closed questions are not the villain; leading ones are. "Is it three area managers?" is closed and gets a real fact for one minute, and the bench scores it so. The rule is about sequence: confirm with a closed question after an open one has shown you the shape of the answer, never before. And the anti-lever carries its own limit: it is not the lowest score on the bench. The technical opener, at 11, is lower, because jargon wastes more minutes than leading questions do. What makes the efficient interview the anti-lever is not its total but what its total hides: 21 looks like a middling meeting, and five of the lines behind it are wrong, the costliest being the one that becomes the product. The bench does not pretend that speed is the sin. Assuming is.