The gate was true on Tuesday
Session a5 got you to general availability with six green lights and a signature. This session starts one week later, when nothing has shipped, nobody has changed anything, and the complaints have started. That is not a hypothetical failure mode. It is the normal life cycle of a deployed bot, and it is the part almost no launch process has a plan for.
Four things change under a frozen deployment 6 min live
"We have not shipped anything" is not an answer to "why did quality drop". Every one of the four causes below happens without you touching the system, and three of them are entirely outside your repository.
LiveWhy the third row is the one that catches good teams3 min▶
Rows 1 and 2 are visible if you look. A provider changes a model and there is a changelog; somebody edits the knowledge base and there is a commit. Row 3 has no artefact at all. Nobody sends an email saying "the question mix has moved eleven points toward billing enquiries".
And row 3 breaks your evidence rather than your bot. The golden set was sampled to match a traffic distribution, and the whole defensibility of the launch number rested on that match. When the distribution moves, the bot may be performing identically on every category and your headline number is still wrong, because it is a weighted average over the wrong weights. That is the quiet version of this failure and it is invisible from inside the eval.
The leadership implication is small and specific: put a recurring item on somebody's calendar to re-check the category mix against live traffic. Not to re-run the eval, just to check whether the sampling assumption still holds. It takes twenty minutes and it is the highest-yield recurring task on this page.
Offline evaluation gates. Online evaluation monitors. 7 min live
After GA you are running three different evaluation activities on three different clocks, and the single most common mistake is treating them as one thing that somebody keeps forgetting to do. They answer different questions, run on different data and fail into different actions.
Rollback is a procedure you rehearse, not a paragraph you write 6 min live
Every gate you set at launch needs a matching trigger that says what would make you undo the launch. Written in advance, a trigger is a decision. Invented during an incident, it is an argument, and the argument always happens at the worst possible moment.
Self-studyWhy "roll back the code" is usually the wrong instruction3 min read▶
Trace a typical regression backwards. Somebody edited a system prompt to fix a tone complaint. Somebody else moved the deployment onto a newer model snapshot because the old one was being deprecated. Retrieval started returning a different top passage because the corpus was reindexed. Not one of those three is a code change, and a git revert undoes none of them.
So the rollback path has to name the things that actually determine behaviour: the prompt version, the model or snapshot identifier, the retrieval index version, and the tool or function schema. If those four are not versioned artefacts with names, you do not have a rollback - you have a search through recent changes conducted while the incident is ongoing.
The leadership test is a single question to the on-call team: if I asked you to put this bot back to how it behaved on Monday, what would you type? If the answer takes longer than a sentence, that is your ops-readiness gate telling you something.
Incidents become tests, and the suite grows itself 6 min live
The compounding asset of a deployed bot is not the model and it is not the prompt. It is the regression suite, and it only compounds if every incident is converted into a permanent case before the incident is closed.
| Stage | What happens | The discipline that makes it stick |
|---|---|---|
| 1 · Detect | Online eval, a rollback trigger, or a human reports a bad output | The trigger already exists and is tied to a gate, so detection is not a judgement call |
| 2 · Contain | Roll back the prompt or model version, or halt the ramp | Contain first, diagnose second. The diagnosis is better with the pressure off anyway |
| 3 · Convert | The failing input becomes a frozen regression case with the expected behaviour written observably | This is the step that gets skipped. An incident that produced no test case will happen again, and the second time nobody will remember the first |
| 4 · Re-gate | The new case joins the blocking suite, and the fix has to pass it before it ships | The suite is only worth anything if it blocks. A regression suite that reports and does not block is a dashboard |
| 5 · Re-expand | Ramp back up through the rungs you came down | Not straight back to 100%. You are re-earning the rungs, on a build nobody has canaried yet |
This loop is OpenAI's stated deployment approach, written as a product process. Release gradually rather than all at once, pair every release with monitoring and the standing ability to pause or roll back, study the failures that real usage produces, convert those failures into evaluations and safeguards, then expand access again. The important word in that sequence is convert. Every organisation studies its failures; the ones whose bots get durably better are the ones where studying a failure produces an artefact that outlives the person who studied it.
The refresh cadence
The suite decays too. A golden set sampled in March is a March artefact, and by June it is answering questions your users have stopped asking. A workable rhythm, drawn from practitioner guidance rather than from any standard:
| Trigger or cadence | What you do | Why this interval |
|---|---|---|
| Every significant change | Error analysis on a fresh batch of traces, not just a re-run of the old suite | A change moves the failure distribution. Re-running the old set tells you about the old failures |
| Every 2 to 4 weeks | A review cycle over 100 or more fresh production traces | The same 100-trace floor that error analysis needs anywhere. Fewer than that and you are reading anecdotes |
| Weekly | 10 to 20 outlier traces: the slowest, the most expensive, the lowest-scored, the escalated | Cheap enough to actually happen, and outliers surface new failure modes far faster than random sampling |
| Stop condition | Stop adding failure categories when roughly 20 consecutive traces produce nothing new | Saturation. Past that point you are labelling for the sake of labelling, and the marginal trace teaches you nothing |
Write six rollback triggers, in the room ★ 8 min · everyone writes
One trigger per gate, each with a number and a time window in it. A trigger without a number is a feeling. Take the six gates from a5 and fill them in for a bot you actually run.
For each gate, complete the sentence: "we roll back if [measured thing] crosses [number] for [duration]." The duration matters as much as the number - "p95 latency over 2 seconds" fires on every traffic spike, while "p95 latency over 2 seconds for fifteen minutes" fires on a real regression.
Mark which triggers halt the ramp and which roll back immediately. Safety is the one that should have no discussion step in it. If every trigger on your list ends in "investigate", you have written an alerting policy, not a rollback policy.
Name who is allowed to fire each trigger at 3am without asking anyone. If that person is you, and you are not on call, the trigger will not fire. Move it to somebody who is awake.
Now the uncomfortable one: ask what you would actually type to roll back. Prompt version, model snapshot, retrieval index, tool schema. If any of those four is not a named artefact you can revert to, write that down as the top item of your ops backlog.
Schedule the rehearsal. An actual timed revert in staging, with somebody holding a stopwatch, on a date in the next fortnight. Put the elapsed time into your launch documentation. That number is the single most credible thing in a readiness review.
Score your operations honestly ★ 7 min · self-assessment
Take one deployed bot and answer these five questions with yes, partly, or no. Then apply the ML Test Score rule: your result is your worst answer, not your average one.
Is anything evaluating this bot on production traffic right now? Not "is it logged" and not "could we". Is there a running job that scores a sample of live outputs and would alarm if the rate moved.
When did the golden set last change? If the answer is "at launch", your evidence is now a historical document. Check when the category mix was last compared against live traffic.
Has anybody rehearsed the rollback? Not read the runbook. Executed it, in staging, timed. If not, gate 6 was green on trust rather than on evidence.
How many of your last five incidents produced a permanent regression case? This is the highest-signal question on the list. Anything below five means your suite is not compounding, and the same incident is queued up to happen again.
Would you know if the model underneath you changed? A pinned snapshot with a deprecation date in the calendar is a yes. "We use the latest" is a no, and it is row 1 of Part 1 waiting to happen.
Where the leader track ends, and what comes next
Six sessions ago the question was why bots fail after launching well. The answer turned out to be almost entirely about evidence rather than about models: what a number proves, who set the bar and when, which six things had to be true at once, and what happens on the Tuesday after.
You now have the decisions. You do not yet have the instruments, and that is deliberate - the leader track was built so you can chair a gate review, grill an eval report and refuse a launch without ever opening a terminal. The builder track is the other half: the ten sessions that construct the golden set, the metrics, the calibrated judge, the three archetype gates, the scorecard and the CI wiring that make everything on these six pages produce a number rather than an opinion.
Try it yourself - this week ◐ 45-60 min total
- Book the rollback rehearsal. A date, a name, a staging environment and a stopwatch. Write the elapsed time into the launch document for that bot. This is the highest-value item on the page and it takes an hour.
- Audit your last five incidents against one question: did each one produce a frozen regression case that still runs today. Report the fraction, not the excuse. Anything under five out of five is a leak in the compounding asset.
- Find out whether anything is currently evaluating any of your bots on production traffic. If the answer is no, you have a launch process and no operations, and that is a one-line finding worth taking to your next review.
- Compare your golden set's category mix against last month's live traffic. Twenty minutes of work. If the mix has moved, your headline launch number is a weighted average over the wrong weights, and no code change caused it.
- Pick your worst answer from Discussion 2 and fund exactly that one thing this quarter. Not the average, not the most interesting - the minimum. That is what the rubric says and it is what production does.
Official sources covered
Every figure and framework on this page traces to a primary source catalogued in this course's source map, verified 13 August 2026. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your team scores well on the ML Test Score's data and model sections, and badly on monitoring. How does the rubric score you?
The minimum rule is deliberate, and it is the same rule the six launch gates use. Production failures arrive through your weakest section, so an average would describe a system that does not exist. It is also why the Google survey figures are worth knowing before you score yourself: no single test in that rubric was implemented by more than 80% of teams, and training and serving skew is among the least implemented of all. A low score is normal. Averaging your way out of it is not.
2 · Your bot passed its gate at 92% three months ago. Nothing has been deployed since. Complaints are climbing. What is the most likely explanation?
A frozen deployment drifts because three of the four inputs to its behaviour live outside your repository. The subtle one is the third: if the user question mix has moved, your bot can be performing identically on every category while your headline number is wrong, because it was a weighted average over a distribution that no longer exists. That is why the launch figure is a snapshot and why continuous evaluation on sampled production traffic is the only thing that turns it back into a claim about today.
3 · Your rollback plan is "revert the last commit and redeploy". What is wrong with it?
Trace a regression backwards and it is usually a prompt edit, a model snapshot change, or a reindexed corpus - none of which a git revert touches. Naming those artefacts and versioning them is half the fix. The other half is rehearsal: the ML Test Score rubric lists safe and fast rollback as a production-readiness test and explicitly says operators should practice doing it. A rollback that has never been executed is a plan. The useful leadership question is simply "what would you type", and if the answer takes longer than a sentence, gate 6 was never really green.