Risk, drift and governance when the eval itself is the problem

The first two lies come from inside the eval machinery itself.

• Gaming the metric (Goodhart's law). When a measure becomes a target, it stops being a good measure. Optimize hard for one number and the system finds a shortcut that wins the number while betraying the goal - padding answers to please a length-happy judge, or refusing more often to protect a faithfulness score. The metric climbs; the product gets worse.
• An untrustworthy LLM judge. Using a strong model to grade answers is how eval scales, but these judges have documented biases. They tend to favor whichever answer they see first (position bias), favor longer answers regardless of quality (verbosity bias), and favor answers written in their own family's style (self-preference bias). A judge with an unaddressed bias produces a confident, precise, and wrong score.

Every offline eval rests on a golden set: a fixed collection of questions with known-good answers. That set is the ruler you measure against, and a warped ruler gives clean, precise, wrong readings.

• Stale. The set was built a year ago. Since then products changed, policies changed, users started asking new things. The system aces the old questions and fails the new ones - and the eval, looking only at the old questions, reports success.
• Skewed. The set over-represents the easy, tidy questions someone found convenient to write and under-represents the messy, ambiguous, high-stakes ones real users bring. A high score on an unrepresentative set is a high score on a test that does not resemble reality.
• The leader question: "How old is the set we grade against, who curates it, and how do we know it looks like what our users actually ask today?"

The fourth lie is the cruelest because it needs no mistake at all. You built an honest eval, set a fair threshold, passed it on launch day - and then the world kept moving. Documents get updated, customer questions shift with the season, the underlying model gets a silent update, a competitor changes the market. Nothing in your system broke; the ground under it moved. An eval run only at launch cannot see this. It reported the truth about a day that has passed. The remedy is to keep measuring on live traffic after launch, which is why this is the bridge into the second half of the session and into the online eval you met in a3.

Both are forms of the world moving, but you catch them in different places.

• Data drift is a statistical shift in the input distribution - the mix of questions coming in changes. A new product launches and suddenly a quarter of queries are about it. You catch this by monitoring the inputs themselves and alerting when their distribution moves past a threshold. The system may still be answering correctly; it is just being asked something new.
• Concept drift is when the relationship between input and desired output changes. The questions look identical to last month's, but the right answer is now different - a policy changed, a price changed, a rule was updated. This is the dangerous one, because the inputs give you no warning. You catch it by watching the quality of the answers on live traffic, not just the shape of the questions.

An LLM judge is how eval scales past what humans can hand-grade, including explaining why a live answer looks wrong. You can allow it - but calibrated and checked, never blindly.

• Calibrate for the known biases. Swap the order of the two answers being compared and require the judge to give a consistent verdict either way - that neutralizes position bias. Control for length so verbosity is not rewarded. Where a reference answer exists, have the judge grade against it rather than on vibes.
• Use a different model to grade than the one under test. A judge from the same family as the system being graded tends to favor its own style. A different model family as the grader removes that self-preference tilt.
• Validate against humans. Keep a small set of human-labeled examples and check that the judge agrees with people on those. If it drifts from human judgment, recalibrate before you trust it at scale.

All of this rolls up into a governance question that is squarely a leadership responsibility, not a technical one: who is accountable for the eval, and how do we know the eval itself is still honest? A trustworthy setup has a named owner for evaluation, a schedule on which the golden set is refreshed with real failures, drift monitoring on live traffic, and a written policy for when a human must sign off rather than a machine. None of that is code. All of it is governance - the standing rules that keep the instrument trustworthy long after the launch excitement fades. Session a6 turns these one-off good intentions into a durable culture.