Your eval governance playbook

The scorecard is a demand, not a form you police. Three usage rules keep it light:

• The team fills it, you read it. Fifteen minutes of reading replaces an hour of interrogation, because the blanks do the interrogating.
• Blanks are findings, not failures. "We have no cost matrix owner" said out loud is progress - it names the exact next meeting. What you are eliminating is the silent blank.
• One page, enforced. A 30-page eval appendix is how weak numbers hide. The one-page constraint forces the eight numbers that matter to stand in the open.

If a slot feels shaky when you try to demand it, here is where it came from:

• Slot 1 (matrix + base rate) - a1, the accuracy paradox: a 96.8%-accurate model on a 3.2% base rate that finds nobody.
• Slots 2-3 (cost matrix, operating point) - a2, where the F1-optimal threshold cost $704 more than the dollar-optimal one, and someone had to choose.
• Slot 4 (judge agreement) - a3, judge skepticism: a judge score is a model output until calibrated against humans.
• Slot 5 (latency + cost budgets) - a4, satisficing metrics: quality subject to speed and price floors.
• Slots 6-8 (golden set, regression status, proxy chain) - a5's three killers, turned from incidents into standing fields.

The scorecard is not new material - it is the six sessions made demandable.

• "Show me the retention data, not the engagement data." Engagement is the proxy; retention is closer to the money. A vendor who pivots back to engagement three times is telling you which number they have.
• "What is your judge's agreement with human labels?" If their quality scores come from an LLM judge, this number must exist. The bar: "over 80%", which equals human-human agreement. "Our judge is GPT-4-class" is not an answer - it is the setup for the question.
• "What happened to your metrics when the base model last changed?" Real discipline produces a before-and-after regression report within a day of any model change. If they cannot show one, they are not measuring - they are hoping.
• "Walk me through one case your system failed last month." Teams with golden sets and production sampling have failure cases catalogued and love talking about them. Demo theater has never seen a failure, which tells you nobody looked.

Rule #39's insight, and the frame to hold every eval claim in: eval score -> online metric -> business goal. Two hops, and each hop can break. The offline eval predicts the online metric imperfectly (hop 1); the online metric serves the business goal imperfectly (hop 2). A benchmark win is a hop-1 event - real, worth having, and two broken links away from revenue.

The discipline is not cynicism - it is precision: celebrate hop 1 as hop 1. "The eval improved, which historically predicts a lift in conversion, which we will confirm in the next release" is a leader's sentence. "The eval improved, ship it and book the value" is how killer 2 gets funded for another quarter.

Questions asked in a meeting can be charmed away; questions in a procurement document cannot. Three clauses worth adding to your next RFP or renewal:

• Evidence clause: "Vendor will provide, for the proposed system, its evaluation methodology including golden set size, grading method, and - where an LLM judge is used - the judge's measured agreement with human labels."
• Change clause: "Vendor will notify us before changes to the underlying model version, and will provide before-and-after regression results for any such change."
• Exit clause: "Evaluation artifacts (golden sets, scoring rubrics, historical results) relating to our deployment are deliverables, portable at contract end."

The third clause is the quiet power move: it makes the concepts yours even when the product is not - the same reason this course teaches taxonomies, not vendor buttons.

Three checks, thirty minutes, four times a year:

• Golden set freshness. Do the 20-50 cases still look like this quarter's real traffic? Users invent new phrasings; the set must follow them.
• Drift check. Sampled production scores vs the launch baseline - is the gap growing? This is a5's killer 3 on a leash.
• Price re-validation. Are the FP and FN costs still true? Margins move, order values move - a2's operating point is only as good as the prices behind it.

And the ownership map, so the review never becomes everyone's job and therefore no one's:

Notice the pattern: it is the skew ladder's three owners again, now with standing responsibilities instead of incident response. Governance is just incident response moved before the incident.

• You write the eval sheets, or your team does: the practitioner track of this course (b1-b10) is the hands-on version of everything you just learned - the live threshold evaluator, the $704 Goodhart demo you can drag yourself, the judge you calibrate from 58% to 88% agreement, and the b10 capstone that writes a full eval sheet.
• You want the LLM eval machinery hands-on: the sibling learn-evals course owns the harness - golden sets as code, RAGAS pipelines, tracing, CI wiring. This course taught you what the numbers mean; that one makes them run.
• The classic-ML mechanics felt distant: learn-intro-ml teaches the sklearn side - training the very Lumen conversion model this course has been evaluating, plus the metrics computed from scratch.

And the standing homework from a5, now upgraded: run the pre-mortem on your flagship project, then bring the eight-slot scorecard to its next review. The first time a blank slot changes a ship decision, this course has paid for itself.