An eval that only runs when somebody remembers is not a gate
There is a specific way this fails, and it is boringly consistent. The eval is built for the launch, it passes, the bot ships, and then a prompt is edited on a Thursday to fix one customer complaint. Nobody re-runs anything, because re-running is forty minutes and the change was small. Six weeks later the bot is measurably worse than the version you signed off, and nothing in the system knows.
Two suites, three triggers 6 min live
b1 split the golden set into a capability suite and a regression suite and promised this session would wire them up. The wiring is the easy part. The design decision is which trigger each one hangs from, because the trigger is what determines the size, the grading rung, and whether the suite is allowed to say no.
| Lane 1 · blocking | Lane 2 · tracking | Lane 3 · production | |
|---|---|---|---|
| Trigger | Every pull request | Nightly, and per release candidate | Continuous, plus a schedule |
| Set | Regression cases + safety canary | Capability suite, hundreds of cases | Sampled live traffic + a fixed drift set |
| Grading | Rule and execution only | Mixed rungs, judge acceptable | Judge and rules, plus user signals |
| Bar | Near 100%; canary exactly 100% | No bar - a trend against baseline | Alert thresholds, not pass marks |
| On failure | The merge stops | A conversation, and a ticket | An alert, and possibly a rollback |
| Grows by | One frozen case per incident | Fresh traces every 2-4 weeks | Nothing - it samples what exists |
LiveWhy the capability suite must never block3 min▶
Every team eventually proposes it: "the capability suite is our real measure of quality, so let us block merges on it too." It is a reasonable instinct and it destroys the pipeline within a month. Three reasons, in order of how quickly they bite:
- It is slow and expensive. Hundreds of cases with judge grading is minutes and dollars per run. Attached to every pull request, it becomes the reason people batch changes, which is the opposite of what you wanted.
- It is designed to fail. A capability suite that passes at 100% is too easy to be informative - b1's point. A suite that is supposed to sit at 70% cannot also be a pass/fail gate without somebody inventing an arbitrary line.
- It is noisy. A judge-graded suite moves a point or two between identical runs. Block on that and you teach the whole team that red means "run it again", which is precisely the habit that later lets a real regression through.
Track it, trend it, and require an explanation when it moves. That is a strong social gate. It is just not a mechanical one.
You buy sharpness by removing noise, not by adding cases 7 min live
This is the statistical constraint that shapes every design decision in Lane 1, and it is genuinely counter-intuitive: the thing that makes a small blocking suite trustworthy is not its size. It is that nothing in it is allowed to have an opinion.
Self-studyThe tooling, and what each one actually gives you in a pipeline4 min read▶
Four capabilities matter for CI, and they are unevenly distributed:
- promptfoo (MIT) exits non-zero when the suite fails, which is all a pipeline strictly needs, and its docs give a pass-rate threshold pattern - their example gates at 95%. Its GitHub Action posts pass/fail comments on the pull request, so the result lands where the change is being discussed rather than in a log nobody opens. Critically,
repeatandrepeat-min-passturn a flaky model-graded test into an honest "m of k runs" assertion instead of an all-or-nothing coin flip. - DeepEval (Apache-2.0) gives you
assert_test()with per-metric thresholds in ordinary pytest files, run as a pipeline step withdeepeval test run. If your team already knows pytest, this is the shortest path from "we have metrics" to "the build goes red". - Langfuse (MIT except
ee/) supports CI gating on regressions alongside its judge and annotation queues, which matters when the same platform is also holding your production traces. - Arize Phoenix ships pre-tested evaluators with published benchmark targets, which is worth knowing about even if you gate elsewhere - it is the closest thing to a public calibration reference. Note the licence: Elastic License 2.0, not plain open source.
Two operational habits are worth more than the choice between them: cache LLM calls so an unchanged case costs nothing on the second run, and run heavy red-team scans nightly rather than per pull request. A red-team suite is Lane 3 work wearing Lane 1 clothes, and putting it on the merge path is how teams end up disabling it.
Three kinds of production evaluation, and one loop 6 min live
Offline evaluation asked "is this version good enough to release". Production asks three different questions, on three different clocks: is it still working now, is it still working the same way it was in March, and can it still survive an attack. None of those can be answered by the set you launched with.
Google's own rubric says the part everybody skips out loud. The ML Test Score lists rollback as a production-readiness test and specifies that it is an emergency procedure operators should practise - not document, practise. In Google's internal survey across teams, no single test in that rubric was implemented by more than 80% of teams, and the scoring rule is deliberately unforgiving: the final score is the minimum across the four sections, not the average. Excellent evals do not compensate for an unrehearsed rollback, in exactly the way a green quality gate did not compensate for a red ops gate in b9. And the practical detail that ruins most first attempts: teams roll back the code and leave the prompt and model version untouched, because those live in a config store nobody put in the runbook.
Wire Beacon's Lane 1 into the pipeline ★ 11 min · everyone builds
Beacon has three bots and one pipeline. The support bot's regression suite and its canary are the sharpest of the three, because they are the only ones where a miss costs money. Build this one; the other two are the same shape with different cases.
Split your existing eval file in two: regression.yaml holds the frozen incident cases and only the rule-graded and execution-graded ones, and capability.yaml holds everything else. If a case cannot be graded deterministically, it does not go in the blocking file. No exceptions, however important the case feels.
Add a third file, canary.yaml, with the handful of cases that must never fail: the approval-before-refund rail from b8, a prompt-injection case, a cross-tenant data case. This runs as its own job, with its own exit code, and it does not contribute to any average anywhere.
Wire both to pull_request. The runner exits non-zero on failure, which is what fails the job - no extra logic needed. Turn on caching so an unchanged case does not pay for the same model call twice; a cached Lane 1 run should finish in well under a coffee.
For any model-graded case you could not eliminate, use repeat with a minimum pass count rather than accepting a flaky all-or-nothing assertion. Three of five is an honest statement about a non-deterministic system; one of one is a lie you will be re-running all week.
Move the capability suite and the red-team scan to a nightly schedule, and have them post their result rather than fail the build. Then delete the manual "run the evals" line from your release checklist. If it is on a checklist, it is not a gate.
Close the loop: incident to permanent case ★ 9 min · everyone writes
Beacon's insight bot mis-stated a revenue figure in an executive summary on a Tuesday. Walk it through the loop now, on paper, so that when it happens for real the path is already worn.
Write the incident as one observable sentence: what the bot did, what it should have done, and how you found out. "It hallucinated" is not that sentence. "It reported Q3 revenue from a query that filtered on the wrong fiscal calendar, and a director caught it in a board pack" is.
Freeze the exact input and the exact expected behaviour into one regression case. One, not a family - families come later from error analysis. Grade it at the lowest rung that catches it, which for a numeric claim is almost always execution rather than a judge.
Ask whether the case belongs in the canary. The test: would you stop a release for this one case alone, ignoring every other number? If yes, it is canary. If you hesitate, it is regression. Very few cases are canary, and the discipline of that question is what keeps the canary meaningful.
Add the rollback trigger this incident should have fired, phrased against a specific gate and threshold - not "if quality drops". Then check whether that trigger was even measurable in production. Usually it was not, and that gap is the actual finding of the incident review.
Schedule the error analysis. Pull 100+ fresh traces in the next two weeks, review 10-20 outliers, and stop adding new failure categories once roughly 20 consecutive traces show you nothing new. That saturation point is your signal to stop looking and start fixing.
Try it yourself - the next two weeks ◐ 90 min total
- Wire one blocking job on one repository, even if the suite has only six cases in it. A six-case gate that runs on every pull request beats a five-hundred-case suite that runs when somebody remembers, and it is the version that will still exist in six months.
- Separate your canary into its own job with its own exit code, and verify it by deliberately breaking one canary case on a branch. If the build stays green, you do not have a canary - you have a slice of an average.
- Rehearse a rollback with a stopwatch, and make sure the thing you roll back includes the prompt and the model version, not only the code. Write the number down. It belongs in the sign-off paragraph from b9.
- Book the error-analysis session into the calendar as a recurring event: 100+ fresh traces every two to four weeks, 10-20 outliers weekly. Unscheduled error analysis does not happen, and error analysis is the only mechanism that keeps your golden set from slowly describing a product you no longer ship.
- Take the last production incident you had and write its regression case today. Then add "the regression case exists" to your definition of done for incident fixes, and watch how much resistance that one line generates. The resistance is the point.
Official sources covered
Every framework and figure on this page traces to a primary source - vendor documentation, published rubrics and peer-reviewed work catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your per-pull-request gate runs 100 cases and you want it to catch a 4-point regression. What do you do?
A gate on the merge path has to stay fast and cheap, which caps n hard, so buying sharpness with volume is not available - that is Lane 2's job. What is available is variance reduction: deterministic grading means a failure is a failure rather than a judge having an off day, and a paired comparison against the same questions the baseline ran cancels the question-level variance, so a smaller difference becomes detectable at the same n. Option C makes the gate noisier, not sharper: you would be trading a missed regression for a stream of false alarms, and false alarms are how teams learn to ignore red.
2 · Your suite reports 99.2% and two of the eight safety-canary cases failed. The pipeline is green. What went wrong?
Eight cases inside a 500-case suite are 1.6% of the score, so two failures move the aggregate by less than half a point and the dashboard stays green while the pipeline merges a version that can issue an unapproved refund. The canary is not a slice of a rate; it is a separate binary gate that blocks regardless of how good the aggregate looks. Give it its own job, its own exit code, and the authority to fail the build alone.
3 · An incident is fixed in production and the bot no longer does the bad thing. Is the incident closed?
Nobody argues against the incident-to-eval loop; it just silently does not happen, because at the moment of an incident everyone is busy shipping the fix and the case is a twenty-minute task with no deadline. Without the case, the same regression can return with the next prompt edit and nothing in the system will know. This is OpenAI's iterative deployment loop in practice - ship gradually, monitor, keep the ability to roll back, and convert real failures into permanent evals - and it is the only mechanism in this whole course that makes your gate get stronger over time instead of staler.
Ten sessions, and the one sentence underneath all of them
You started this track with a bot and a number that felt roughly true. You end it with something a good deal less comfortable and a good deal more useful: a system that produces numbers you can defend, keeps producing them after you stop paying attention, and gets stronger every time something goes wrong.
| What you have now | Where it came from |
|---|---|
| A golden set sampled from real traffic, in real proportions, with an adversarial and unanswerable slice | b1 |
| The habit of grading at the cheapest rung that settles the case, and quality and cost metrics with real formulas | b1, b2, b3 |
| A judge calibrated against human labels, or an honest written note that it is not | b4 |
| Three gates for three archetypes, because containment, execution accuracy and a validity cascade are not the same test | b5, b6, b7 |
| A trajectory spec that grades the path, and a pass^k number for the workflows you actually ship | b8 |
| A six-gate conjunctive scorecard and a sign-off paragraph with your name on it | b9 |
| Two suites on two triggers, a canary that blocks alone, rollback triggers, a refresh cadence, and the incident loop | b10 |
That sentence is why the interval mattered in b1 and a4, why the threshold had to be set before the run, why an uncalibrated judge disqualifies a result no matter how large the set, why pass^k belongs next to pass@1 on the same table, and why a gate that only runs when somebody remembers is not evidence about the system you are actually operating. It is also the sentence that will make you unpopular in exactly one meeting and trusted in every meeting after it.