Drift and silent changes: the ground truth moved and nobody deployed

The best available teaching artifact for output-quality drift is a primary-source postmortem in which a model provider documents how hard its own degradation was to detect. Three overlapping issues, reported by users, and the internal benchmarks stayed green.

• Scale, from the postmortem itself: at peak, 16% of Sonnet 4 requests were affected in the worst hour, and approximately 30% of Claude Code users who made requests during that period had at least one message routed to the wrong server type.
• Platform asymmetry: peak impact on Bedrock was 0.18% and on Vertex under 0.0004%. If you multi-home, one platform's telemetry does not represent the others, and the platform you happen to dashboard is the one whose numbers you will believe.
• Why detection failed: the benchmarks "simply didn't capture the degradation users were reporting", partly because the model recovers from isolated errors, and because the bugs produced different symptoms on different platforms at different rates.

Take exactly two lessons. First: user reports are a signal. They are noisy, unstructured and frequently right. Second, and this is the hard one: "our evals are green" is not a rebuttal to a user report. It is a statement about a distribution you constructed, and drift is precisely the case where your constructed distribution has stopped resembling production traffic. When those two disagree, the honest next step is to sample real traces from the complaining segment, not to re-run the suite.

Quality scorers are expensive to build, need calibration, and drift themselves. Meanwhile your users are already scoring every answer, in a currency you can measure without any judge at all:

• Retry rate. The same conversation id asking a near-identical question within a minute. One person deciding the answer was useless, recorded as a fact rather than an opinion.
• Escalation to human. The strongest signal available in any product with a support path, and usually already instrumented for a different reason.
• Abandonment. A session that ends immediately after an answer, with no follow-up and no action taken on the result.

These are lagging by minutes rather than by a quarter, they cost nothing to compute, and they are honest in the way judge scores are not, because nobody had to define what "good" means. Their weakness is diagnosis: they tell you something got worse and nothing about what. So the sequence that works is: downstream behaviour alerts you, sampled traces explain it, and a scorer gives you the trend line afterwards. Teams that build in the opposite order spend a quarter on a judge before they have anything watching production.

Pinning is necessary and it is not sufficient, and it is worth being precise about the residue so you do not oversell it internally:

• The clock still runs. A pinned snapshot has a lifecycle: Active, then Legacy, then Deprecated, then Retired. The commitment is at least 60 days' notice, and requests to a model past its retirement date fail. Pinning converts a silent quality change into a dated hard failure, which is a large improvement and not an exemption.
• Serving-side variation persists. Identical kernels are run-to-run deterministic but not batch-invariant, and server batch size varies with other users' load. The other concurrent users are not an input to your system but they are a nondeterministic property of it, which is why 1000 identical prompts at temperature 0 produced 80 distinct completions.
• Your reproducibility levers may be gone. On Opus 4.7 and later, temperature, top_p and top_k are deprecated and return 400 on non-default values, so "just set temperature to 0 and replay it" is not available as a strategy.

Which is the whole argument of this course restated: because you cannot reproduce a production trace by replaying it, you must capture enough of it at the time it happens.

Until recently, eval scores lived in one system and traces lived in another, and joining them was somebody's weekend project. That is changing: the GenAI semantic conventions now define an evaluation signal, which means a score can ride the same pipeline as the span it describes.

• The event is gen_ai.evaluation.result, carrying gen_ai.evaluation.name, gen_ai.evaluation.score.value, gen_ai.evaluation.score.label and gen_ai.evaluation.explanation.
• The practical consequence is the one that matters for this session: a drifting score is now a time series sitting next to the trace that produced it, so "quality dropped" and "here are forty traces from the drop" become the same query instead of two systems and a spreadsheet.
• The usual caveat applies with full force. Like everything else in these conventions it is Development status with no tagged release, so check the current spelling in the repository before you build a dashboard on it.

OpenInference reaches the same place from the other direction, with EVALUATOR as one of its ten span kinds. Different dialect, same convergence: the evaluator is becoming part of the trace rather than a separate report.