Builder session 8 · Regression eval suites

Part 0

A metric you do not gate on is decoration

Hit Rate, faithfulness, answer correctness - all of them tell you how Recall is doing right now. None of them stop a bad change from shipping unless something acts on the number. Engineers already solved this problem for code: a test suite that runs on every change and a CI job that refuses to deploy when tests fail. An eval suite is the same machinery pointed at model behaviour. Today we make Recall's evals block a regression, not just report one.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end

★ What you build today A promptfoo config that runs Recall's golden questions with real assertions, and a bash CI gate that parses the failures and exits non-zero when the pass rate drops below your threshold - the wiring that makes an eval a gate instead of a dashboard.

Part 1 · covers the eval suite as a test suite

The eval suite is a test suite 9 min live

Reframe the whole track in one sentence: your golden set plus your metrics is a suite of tests, and every change to Recall is a commit that must pass it. A new system prompt, a swapped model, a tweaked retrieval k - each triggers a full run of the suite, and the result is a pass or a fail, not a paragraph of vibes. promptfoo is a config-driven runner built for exactly this: you declare prompts, providers, and tests, and it runs the matrix.

🔍 Click to zoom - a change runs the suite, a pass/fail gate ships or blocks it

LiveAssertions: contains, similar, llm-rubric4 min▶

A test needs an assertion - the thing that turns an output into a pass or a fail. promptfoo gives you a ladder of them, from cheap and strict to smart and fuzzy, and you pick per test.

contains / equals - the output must contain (or exactly equal) a string. Cheap, deterministic, perfect for "the refund answer must mention 5 business days".
similar - an embedding assertion: the output must be semantically close to an expected answer above a similarity threshold. Forgives phrasing, catches meaning drift.
llm-rubric - an LLM grades the output against a rubric you write in plain language ("the answer is polite and cites a policy"). This is the LLM-as-judge from b4, wired as a test assertion.

The suite runs on every change The point is not to run it once. It is that a single command re-scores the whole golden set against every assertion, so any change - a prompt edit, a model swap - gets the same battery of tests before it ships. Same discipline as running the unit tests before you merge.

Self-studyA promptfoo config for Recall4 min read▶

Everything lives in promptfooconfig.yaml: the prompts under test, the providers (models) to run them on, and the tests - each with its variables and its assertions. This is the whole suite as one declarative file.

YAML · promptfooconfig.yaml# Recall regression suite prompts: - "Answer the support question using only the context.\nQ: {{question}}" providers: - openai:gpt-4o-mini - anthropic:claude-haiku-4-5 tests: - vars: question: "How long do refunds take?" assert: - type: contains value: "5 business days" - type: similar value: "Refunds reach the original method in 5 business days." threshold: 0.8 - vars: question: "How do I reset my password?" assert: - type: llm-rubric value: "Tells the user to use the Forgot password link and mentions a reset email."

Run the whole thing with one command. promptfoo executes every prompt against every provider for every test, applies the assertions, and prints a pass/fail matrix.

bash · run the suite# run every test against every provider promptfoo eval -c promptfooconfig.yaml # open the results in the local viewer promptfoo view

Pick the cheapest assertion that catches the bug Reach for contains or equals first - they are free and deterministic. Use similar when phrasing varies. Save llm-rubric for judgements a string match cannot make, since it costs a model call per test. A suite that is all rubric is slow and non-deterministic; a suite that is all string-match misses meaning drift.

Part 2 · covers the CI threshold gate

The CI threshold gate 9 min live

Running the suite is half the job. The other half is making its result block a deploy. In CI that means one thing: an exit code. A job that exits 0 lets the pipeline continue; a job that exits non-zero stops it. So the gate is simple - run the suite, compare the pass rate to a threshold, and exit 1 when the threshold is breached. The pipeline does the rest.

🔍 Click to zoom - eval runs, pass rate vs threshold, exit code gates the deploy

LiveThe bash gate: parse, compare, exit5 min▶

promptfoo can write its results as JSON. A short script reads the failure count out of that JSON, computes the pass rate, checks it against your threshold, and sets the exit code. CI reads the exit code and gates the deploy. Note the escaped comparison and redirect operators - bash uses -gt and -lt for integer comparison and > to redirect output.

bash · CI gate on the pass rate#!/usr/bin/env bash set -euo pipefail THRESHOLD=95 # require >= 95% of tests to pass # run the suite and write results to JSON promptfoo eval -c promptfooconfig.yaml -o results.json # pull totals out of the JSON with jq FAILURES=$(jq '.results.stats.failures' results.json) SUCCESSES=$(jq '.results.stats.successes' results.json) TOTAL=$(( SUCCESSES + FAILURES )) # integer pass rate, 0-100 PASS_RATE=$(( SUCCESSES * 100 / TOTAL )) echo "pass rate: ${PASS_RATE}% (${FAILURES} failures)" > eval-report.txt # gate: block the deploy if we dropped below the bar if [ "$PASS_RATE" -lt "$THRESHOLD" ]; then echo "REGRESSION: ${PASS_RATE}% < ${THRESHOLD}% - blocking deploy" exit 1 fi # also block on any hard failure count you refuse to tolerate if [ "$FAILURES" -gt 0 ]; then echo "warning: ${FAILURES} test(s) failed" fi echo "eval gate passed" exit 0

Real world A regression caught before ship A teammate swaps Recall's model to a cheaper one to cut cost. Locally the refund answer still reads fine, so the PR looks safe. CI runs the suite: the contains "5 business days" assertion now fails on four questions because the new model rounds to "about a week", and the llm-rubric password test fails twice. Pass rate drops to 88%, below the 95% threshold, the gate script exits 1, and the pipeline blocks the merge with eval-report.txt attached. The cost saving was real; the silent quality drop would have shipped without this gate. The exit code caught it before a single user saw the worse answer.

Set the threshold from a known-good baseline Do not pick 95% from the air. Run the suite on the current shipped version, read its pass rate, and set the gate at or just below that. The gate's job is to catch a drop from where you already are - a change that makes Recall worse than today - not to enforce a number you have never actually hit.

Self-studyThe same gate from LangSmith's dataset side4 min read▶

promptfoo drives the suite from a YAML config. LangSmith drives the same discipline from a stored dataset: you register a dataset of examples, define evaluators, and call evaluate(). The result is a run you can compare against previous runs and threshold in exactly the same way.

Python · LangSmith evaluate() over a datasetfrom langsmith import evaluate def recall_target(inputs: dict) -> dict: # your system under test: returns Recall's answer for one example return {"answer": recall_answer(inputs["question"])} def correctness(run, example) -> dict: # an evaluator: compare run output to the reference on the example ok = example.outputs["answer"].lower() in run.outputs["answer"].lower() return {"key": "correctness", "score": 1.0 if ok else 0.0} result = evaluate( recall_target, data="recall-golden-set", # a dataset registered in LangSmith evaluators=[correctness], ) # read the aggregate score, compare to your threshold, gate the same way

Honesty note - CI patterns are concept-level The promptfoo config and assertions are documented and stable. The exact CI wiring - which JSON keys, which jq path, how evaluate()'s result is turned into an exit code - varies by version and by your pipeline (GitHub Actions, GitLab, and so on). Treat the bash gate here as the shape of the solution, not a copy-paste for your setup. The durable idea is: run the suite, reduce it to a pass rate, threshold it, and let the exit code gate the deploy. OpenAI's evals framework follows the same run-and-threshold shape from its own harness.

Build-along · take it further

Define Recall's gate ★ 10 min · design it

A gate is a decision made in advance. Write Recall's now, before a risky change forces the question in a hurry.

Set the threshold. Decide the pass rate below which a change to Recall must be blocked. Base it on today's suite result, not a wish. Write it as one number with one sentence of justification.

Pick assertion one. Choose a strict, cheap assertion to gate on - a contains on a fact that must appear (a specific policy number, a refund window). Name the question it guards.

Pick assertion two. Choose a smarter assertion - a similar or llm-rubric that catches meaning drift a string match would miss. Write the rubric or the expected answer in one line.

Trace the exit code. In one line each: what does the pipeline do when the gate exits 0, and what does it do when it exits 1? If you cannot state both, the gate is not wired yet.

★ Recall's gate after b8 Recall's eval suite is now a test suite with teeth. A change that drops the pass rate below your threshold makes the gate exit non-zero and blocks the deploy - the same way a failing unit test blocks a merge. Recall's quality can no longer regress quietly. In b9 we add tracing and observability so that when the gate does fire, you can see exactly which step of a run went wrong.

Homework

Before session b9 ◐ 50 min total

Write a promptfooconfig.yaml with at least four of Recall's golden questions and one assertion of each type: contains, similar, and llm-rubric. Run promptfoo eval and read the matrix.
Adapt the bash gate to your own JSON output: confirm it exits 1 when you deliberately break one test, and exits 0 when you fix it. Verify with echo $? after each run.
Change Recall's prompt in a way you expect to help, re-run the suite, and check whether the gate agrees - a reminder that intuition and the number can disagree.
Read: promptfoo config reference and CI/CD guide; skim the LangSmith evaluate() and OpenAI evals docs for the same run-and-threshold shape.

Source material

Official sources covered

Taught from official docs. This page covers promptfoo's config and assertions in full; the CI wiring and the other runners are shown at concept level.

✓promptfoo · config + assertions (contains, similar, llm-rubric) + CI/CDParts 1-2 · promptfooconfig.yaml, promptfoo eval, JSON output, exit-code gate

◐LangSmith · evaluate(target, data, evaluators)Part 2 · same discipline from a stored dataset; wiring is concept-level

◐OpenAI · evals frameworkNamed · same run-and-threshold shape from its own harness

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · In a CI gate, what actually blocks the deploy when the eval fails?

CI gates on the exit code: 0 lets the pipeline continue, non-zero stops it. The gate script exits 1 when the pass rate breaches the threshold, and that exit code is what blocks the deploy.

2 · Which promptfoo assertion uses an LLM to grade the output against criteria you write?

llm-rubric hands the output to an LLM with your plain-language rubric - the LLM-as-judge wired in as a test. contains is a string match; similar is an embedding-distance check.

3 · Why set the pass-rate threshold from the currently shipped version's score?

The gate's job is to catch a regression - a change that makes the system worse than today. Anchoring the threshold to the known-good baseline makes it a real drop detector rather than an unreachable target.

Builder session 8 cheat sheet · pin this

Eval suite = test suiteYour golden set + metrics is a suite of tests; every prompt/model/retrieval change must pass it.

promptfoo configpromptfooconfig.yaml declares prompts, providers, and tests (vars + assert). Run with promptfoo eval.

Assertion laddercontains/equals (cheap, strict) → similar (embedding, forgives phrasing) → llm-rubric (LLM judge, smart, costly).

Cheapest that catches itPrefer string matches; escalate to similar, then llm-rubric only when needed. All-rubric suites are slow and non-deterministic.

Exit code = the gateCI reads the exit code: 0 continues, non-zero blocks. The gate script exits 1 when the threshold breaks.

Parse then compareEmit JSON, read failures with jq, compute pass rate, compare to threshold (e.g. -lt), exit accordingly.

Threshold from baselineSet the bar (e.g. >=95%) from the currently shipped score so the gate catches a real drop, not a wish.

Same shape everywhereLangSmith evaluate(target, data, evaluators) and OpenAI evals do the same run-and-threshold discipline.