The triage question

Google's own engineering guide for machine learning opens with a rule that most people have never read past the title of. Rule number one, verbatim: "Don't be afraid to launch a product without machine learning." The same guide adds that if you think ML will give you a 100 percent boost, a heuristic gets you 50 percent of the way there.

That is Google, the company with the most ML capacity on earth, telling its own engineers to reach for the simple thing first. The reason is not modesty. It is that the simple thing is verifiable, cheap, and already understood by everyone who has to trust its output.

• Deterministic work - the logic can be written down, verified once, and re-run forever. Metrics, thresholds, reconciliations, filters. The answer is checkable against the data.
• Pattern work - the rule cannot be written down, but history contains it. Churn, forecasts, anomalies. A trained model learns the rule you could not articulate.
• Judgment work - unstructured language, synthesis, generation. Theming tickets, drafting, extracting meaning from messy text. No table contains the answer.

Three kinds of work, three tiers of tooling. Most routing mistakes are one kind of work sent to another kind's tool, in either direction - metrics sent to a language model, or language stuck in keyword counters.

Daily active users. Revenue by plan. Cards that failed three times. Every one of these has a definition a human wrote, which means a human can write the query, someone can check it against the data once, and from then on the answer is correct by construction. There is nothing for a model to be intelligent about.

The test in practice: if you can imagine writing the acceptance test, the logic is writable. "The DAU number must equal the count of distinct users with an event yesterday" is an acceptance test. "The summary must capture what customers meant" is not.

The second question catches the failure the first one cannot. If finance counts revenue one way and product counts it another, no tool on any tier fixes that. A dashboard shows the disagreement faster. An LLM papers over it with fluent prose, which is worse, because now the disagreement ships inside a confident paragraph.

Alignment is a meeting, not a model. It is also the cheapest step in the whole pipeline: one hour of arguing about the definition of "active" saves every downstream tool from computing the wrong thing correctly.

A verified query is not "probably right". It is right in the way a proof is right: the logic was checked against the data once, and the same logic runs every time. That property - verifiability - is what lets a CFO sign the number without re-deriving it.

Probabilistic tools can be excellent, calibrated, and worth every cent. What they cannot be is correct by construction. That is not a criticism, it is a category difference, and the whole discipline of this course is refusing to blur it.

The MIT Project NANDA report from July 2025 found that despite 30 to 40 billion dollars of enterprise investment, 95 percent of GenAI pilots showed no measurable profit-and-loss impact. That figure moved markets, and it is usually quoted as proof the technology does not work.

Its own lead author says the opposite. The divide is a learning gap in organizations: the roughly 5 percent that succeed pick one pain point, execute well, and partner smartly. The same report documents a thriving shadow economy of employees quietly succeeding with personal AI tools while the official pilot stalls. The honest reading: pilots fail to cross into the P&L, and the cause is selection and execution, not model capability. Worth knowing: the report is pre-publication and not peer-reviewed, which is exactly the kind of caveat a leader should attach when quoting it.

The RAND report is careful in a way its quoters are not: it says "by some estimates, more than 80 percent of AI projects fail - twice the rate of failure for information technology projects that do not involve AI". RAND cites the estimate; it did not measure it. What RAND did measure, across 65 experienced practitioners, is the ranked causes in the figure above.

The strongest published skeptic case is Goldman Sachs' 2024 report "Gen AI: Too Much Spend, Too Little Benefit?", where Jim Covello argues that roughly a trillion dollars of capex has "little to show for it so far" and that AI "isn't designed to solve the complex problems that would justify the costs". Read it knowing it is a debate: the same document carries Goldman analysts arguing the optimistic side. The takeaway for a router is neither camp's headline - it is that the spend is real, the returns are concentrated, and routing decides which side of the concentration you land on.

The instinct this course trains has a long pedigree. The Analects records Confucius, arriving in a small town governed with elaborate ceremonial music, asking with a smile: 割鸡焉用牛刀 - why use an ox-cleaver to kill a chicken? The modern version is the one on this course's masthead: do not use a nuclear weapon to kill a mosquito.

Notice what the proverb is not saying. It is not anti-cleaver. When there is an ox, bring the cleaver, and bring the best one you can afford. Session 4 is entirely about recognizing the ox. The failure mode is unmatched scale in either direction: the cleaver on the chicken wastes the cleaver, and the paring knife on the ox ruins the knife and leaves the ox annoyed.

• The capable tool is already open. Once a team has an LLM integration, marginal workloads flow to it by convenience, not by fit. Convenience is a real benefit; it is just rarely priced against the verification cost it drags in.
• Demos reward the top rung. An agent doing a metric lookup looks impressive in a demo and indefensible in an audit. The demo audience never sees the eval set that now has to exist forever.
• Nobody gets promoted for a WHERE clause. Incentives tilt toward visible sophistication. A routing playbook makes the boring-but-right choice legible as a decision, which is the only way it survives review season.