The three project killers

How it looks: the team ships on vibes. Someone tries the system on a few examples, it "seems good," it launches. From then on every prompt tweak, model swap, and vendor upgrade is a gamble - nobody can say whether the change helped or hurt, so regressions surface the worst possible way: as customer incidents.

The mandate: before the system is built, 20-50 real cases must exist - actual inputs your users will send, with what "good" looks like written down. This is Anthropic's eval-driven development: write evals for capabilities before building them, the way test-driven development writes the test first. And their grading advice is a relief for busy teams: "prioritize volume over quality" of graded items - many rough judgments beat three perfect ones.

The price: roughly one workshop hour with the people who know the real cases. That hour is the single highest-ROI hour in any AI project - it converts every future change from a gamble into a measurement.

How it looks: the dashboard climbs and the business does not. F1 is up for a quarter, finance is flat. The judge score is up, and users are churning on latency the judge never sees. This is the a2 lesson at program scale - Goodhart's law: when a measure becomes a target, it stops measuring. You met the miniature in a2's live demo, where the F1-optimal threshold cost $704 more than the cost-optimal one, and the dollar-best threshold had the worst-looking F1 on the card.

The mandate: every eval sheet declares its prices (what a false positive costs, what a false negative costs) and names the one business KPI its metric is supposed to move. No prices, no funding. This is also Ng's doctrine made governance: ONE optimizing metric so iteration is fast, with the rest as satisficing floors - quality subject to latency and cost budgets, not quality in a vacuum.

The tell in a meeting: ask "if this metric goes up 10 points, what line item moves?" A confident answer means the wiring exists. A pause means you are funding a proxy.

How it looks: the system passed every eval at launch. Then the data drifted, the vendor silently upgraded the underlying model, and users found phrasings nobody tested. Six months later someone says "the AI stopped working" - and nobody can say when, because nothing was measuring in between. A launch-day eval is a photograph; the world is a movie.

The mandate: two eval families, run continuously. Capability evals track what you are trying to make the system do - they start low and climb. Regression evals guard what already works - Anthropic's rule is that regression suites hold at ~100%, and any dip blocks the change. Then wire evals into CI so nothing ships unmeasured, sample and score production traffic so drift shows up in weeks not quarters, and pin model versions so a vendor upgrade is a decision you make, not one that happens to you.

The three killers are usually presented as separate risks, but in failed projects they arrive as a chain:

• Killer 1 enables killer 2. Without a golden set, the only numbers a team can report are generic proxies - benchmark scores, judge averages, F1 on stale data. Proxies unwired to money are all that exists, so proxies get optimized.
• Killer 2 masks killer 3. A proxy that keeps climbing looks like health. Drift hides comfortably underneath a rising dashboard - the number goes up while the product quietly gets worse for real users.
• Killer 3 re-arms killer 1. Once "the AI stopped working," trust in measurement collapses, the team reverts to shipping on vibes, and the cycle restarts with less budget.

Which is why the mandates come as a set of three, not a menu. If your team wants the hands-on version: the practitioner track covers the golden-set and drift machinery in b9, and the Goodhart demo - with a button that visibly loses $704 - in b5.

The ladder turns a vague dispute into three checkable questions:

• "Does it score much better on training data than on held-out data?" If yes: overfitting. The DS team tightens the model. Classic, well-understood, theirs to fix.
• "Does yesterday's holdout score hold on today's fresh data?" If no: drift. Nobody coded a bug - the customers changed. The data owner refreshes the training window and the golden set.
• "Does the same example get the same score offline and in live serving?" If no: this is the sneaky one. Rule #37 calls the next-day-vs-live gap an engineering bug - a feature computed differently in the serving path, a stale cache, a version mismatch. No amount of modeling fixes it, because it is not a modeling problem.

Three gaps, three owners. The leadership failure mode is sending all three to the DS team - two of the three are not theirs.

All three defuse moves - the golden set, the priced eval sheet, the continuous loop - share one budget implication: eval infrastructure is not optional scope. It is the product's immune system. A body without one does not feel sick; it just dies of the first infection.

Practically, that means the project plan you approve should show eval work as a first-class line: hours for the golden-set workshop before build, the CI eval gate before launch, and sampled production scoring after. When a team offers to "add evals later to hit the date," they are proposing to ship without an immune system - and later, famously, never comes.

Status meetings leak the killers in ordinary sentences. Five to catch:

• "We'll add evals after launch to hit the date." Killer 1, stated as a plan. The golden set costs an hour; the date argument never survives that fact.
• "The demo went great." Offered as evidence, this is killer 1 in dress clothes - a demo is a handful of curated cases, not a golden set.
• "F1 improved again this sprint." With no dollars or business KPI attached, that is killer 2's soundtrack. Ask what line item moves.
• "We just use whatever model version the vendor serves." Killer 3, pre-armed - the vendor's next silent upgrade becomes your unmeasured incident.
• "It worked offline." Not a killer by itself - it is your cue to run the skew ladder and route the gap to its owner.

Pick the AI project you care most about and write one page - three questions, honest answers:

• Does a golden set exist? Not "we test it" - a named artifact with 20-50 real cases and expected outcomes, that anyone can run. If no: killer 1 is armed. The fix is one workshop hour.
• Who owns the prices? Can anyone state what a false positive and a false negative cost, and which business KPI the eval metric is wired to? A name, not a shrug. If no: killer 2 is armed, and every reported improvement is unverifiable.
• When did an eval last run against production? A date, from monitoring - not a launch memo. If the answer is "at launch," killer 3 is armed and counting.

Then rank the three by proximity and bring the page to a6 - the governance playbook session turns your pre-mortem into a standing scorecard and a review cadence.