Evaluation is a habit or it is nothing
Almost every team can run one evaluation. Very few make it a standing part of how they build. The gap between the two is not technical - it is cultural, and culture is set by what leaders reward, schedule, and fund. An eval that happens once, heroically, before a launch, is a photograph. An eval woven into how the team builds is a heartbeat. This session is about installing the heartbeat: writing the eval first, growing the golden set, automating the gate, and naming who owns it.
Eval-driven development 9 min live
The single most important shift is when the eval gets written. In a mature team, the eval comes first - you define what success means before you build the thing, exactly as Anthropic advises. Then build, measure, ship only past the gate, monitor, and feed every real failure back into the golden set. It is a loop, and it never fully stops.
LiveEval before build - define success first3 min▶
The instinct most teams have is to build the thing, get it working, and then - if there is time - test it. Reverse that. Anthropic's guidance to builders is to define your success criteria first, then create empirical evaluations against them, and only then build toward those numbers. Writing the eval first forces the hard conversation early: what does good actually mean here, and how would we know? A team that cannot write its eval does not yet understand what it is building.
- The eval is the spec. When success is defined as measurable criteria up front, "done" stops being a feeling and becomes a threshold. Everyone builds toward the same bar.
- It surfaces disagreement early. Writing the eval first is where you discover that two stakeholders meant different things by "good". Far cheaper to find that on day one than after launch.
- The leader's move: ask to see the eval before you approve the build, not after. "What is the test this has to pass, and did we write it before we started?"
LiveThe golden set as a growing asset3 min▶
The golden set - your collection of test questions with known-good answers - is the most valuable thing an eval culture builds, and it should never stop growing. Anthropic advises prioritizing the volume of automated evaluation cases over a handful of hand-graded ones. The best fuel for that volume is free and arrives every day: real production failures.
- Every real failure is a gift. When the system gets something wrong in production, that case, with its correct answer, gets added to the golden set. The system can never quietly regress on that failure again - the eval now guards it forever.
- The set compounds. A golden set that grows with real failures gets more representative of reality every month, while a set frozen at launch drifts further from it. This is the antidote to the stale-set risk from session a5.
- Volume beats artistry. A large set of automated cases catches more than a small set of lovingly hand-graded ones. Breadth is the goal.
The failure log that became the moat. A team (anonymized) made one rule: every customer-reported wrong answer became a new eval case by end of week, with the verified correct answer attached. Within a year their golden set had grown tenfold, drawn entirely from real misses. Regressions on old bugs dropped to near zero, because the eval now remembered every mistake the team had ever made.
Self-studyAutomate it, and give it an owner2 min read▶
Two things quietly kill an eval culture: manual effort and diffuse ownership. If running the eval is a person's occasional side task, it will lapse the first busy week - and busy weeks are exactly when a risky change ships. The fix is to automate it so the eval runs on every change without anyone remembering to trigger it, and to give the whole discipline a named owner with a schedule. Not "the team" - a person, with a cadence: the golden set is refreshed monthly, drift is reviewed weekly, the gate runs on every release. Ownership plus automation is what turns good intentions into a habit that survives turnover, deadlines, and enthusiasm running out. This is the least glamorous slide in the course and the one that decides whether the other five sessions matter.
The roadmap and where this goes 8 min live
Eval culture is not all-or-nothing - it is a ladder teams climb. Most start at vibes, then add an offline gate, then guard every change with a regression eval, then watch production, and finally reach a culture where eval leads. Knowing your rung tells you the next one. And the ground is shifting: agents are raising the stakes on everything you have learned.
LiveThe maturity ladder and the roles it needs3 min▶
Each rung is a real, fundable step - and each needs people, not just tools.
- Vibes to offline gate. The first climb is from "it seemed fine" to a measured threshold before launch. This is where most of the value arrives, and it needs someone to own the golden set.
- Offline gate to regression in CI. The eval now runs automatically on every change, so nobody has to remember. This needs an engineer to wire it into the release process once.
- Regression to online monitoring. You start watching live traffic for the drift you met in a5. This needs someone accountable for reviewing the monitor on a cadence.
- Online to eval-driven culture. The top rung is where evals are written first and failures feed the set by default. This needs leadership - it is a habit, and habits are set by what leaders reward.
LiveAgents raise the stakes3 min▶
Everything in this course was about systems that answer a question. The frontier now is agents - systems that take actions, call tools, and pursue multi-step goals with real autonomy. More autonomy means more to evaluate, and the surface grows fast.
- New things to measure. Beyond answer quality, agents introduce tool-call accuracy (did it use the right tool correctly), agent goal accuracy (did it actually achieve what the user wanted), and topic adherence (did it stay in bounds across many steps). These become new columns on the scorecard.
- Failures compound across steps. A small error early in a ten-step task can cascade. Evaluating only the final answer misses where things went wrong - you increasingly have to evaluate the path, not just the destination.
- The leader takeaway: the discipline you learned here does not shrink as systems get more capable - it grows. This is a preview, not a lesson; the depth belongs to a later course.
Self-studyA warm close - where you go next2 min read▶
You started this course with a simple, uncomfortable truth: a model always answers, fluently, whether it is right or wrong, and only evaluation tells the two apart. Six sessions later you can demand a scorecard, read it without a data-science degree, tell offline from online, spot the four ways an eval lies, and install the culture that keeps it honest. That is a leader's fluency in evaluation - not building the tests, but knowing what to require and how to read the answer. If you want to see where the answers come from, the systems being evaluated here are often retrieval-based, and the RAG course is the natural companion to this one. And when you are ready to see how these systems get built and chained into the agents we just previewed, the LangChain course picks up the thread. Thank you for spending these six hours making measurement a habit. It is the least glamorous discipline in AI and the one that decides whether any of it can be trusted.
From heroics to habit. A team (anonymized) that spent a year at "vibes" adopted just two rules from this course: write the eval before the build, and turn every production failure into a permanent test case. They did not buy new tools. Eighteen months on, shipping an AI change felt as routine as shipping any other software - because the gate was always there, and it was always current.
Place your org on the ladder ★ 12 min · pen and paper
Be honest about which rung your organization actually stands on today for one real AI system - then name the single next rung and what it would take to reach it.
Read the five rungs - vibes, offline gate, regression in CI, online monitoring, eval-driven culture - and mark where this system genuinely sits. Judge by what happens by default, not by what happens heroically once.
Write the evidence for your placement. "We have a gate but it only runs when someone remembers" is rung 2 reaching for 3, not rung 3.
Name the next rung and the one concrete thing that would get you there - wiring the eval into the release process, or assigning someone to review the drift monitor weekly.
Estimate the cost of that one step in people and time. It is almost always smaller than leaders fear - and the return is every future launch getting safer.
Assign an owner and a cadence ★ 8 min · make it real
An eval culture dies without a named owner and a schedule. For one real system, assign both - specifically enough that it would actually happen.
Name a single person as the eval owner for this system. Not a team, not a rotation with no lead - one accountable name.
Set the cadence for each recurring task: how often the golden set is refreshed with new failures, how often the drift monitor is reviewed, and when the gate runs (ideally on every release, automatically).
Write the one rule that keeps the golden set growing - for example, "every production failure becomes an eval case with its correct answer by end of week".
Decide how the owner reports up: a one-line monthly note to you on the five scorecard numbers and any drift. Make the habit visible to leadership so it stays funded.
Carry it forward ◐ 30 min total
- Finish placing your top system on the maturity ladder, name the next rung, and put the one concrete step to reach it on an actual roadmap with a date.
- Assign a named eval owner and a written cadence for one real system this week - and tell that person they own it.
- Adopt the one rule that grows the golden set: every production failure becomes a permanent eval case. Announce it.
- Optional: switch to the builder track's b1 to watch these ideas run in code, or open the RAG course to see how the systems being evaluated actually retrieve their answers.
- Do we write the eval before we build, or after - and could we define success as measurable criteria up front for our next project?
- Does our golden set grow from real production failures, and when did we last add a case from a customer-reported miss?
- Does our eval run automatically on every change, or only when someone remembers to trigger it?
- Who is the single named owner of evaluation for each system, and what is their cadence for refreshing the set and reviewing drift?
- Which rung of the maturity ladder are we honestly on, and what is the one next rung we should fund this quarter?
Official sources covered
The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · In eval-driven development, when should the eval be written?
Anthropic advises defining success criteria first, then creating empirical evaluations against them, then building. Writing the eval first turns "done" into a threshold and surfaces stakeholder disagreement early.
2 · The best fuel for a golden set that keeps growing is...
Every real failure is a free, perfectly representative test case. Feeding failures back in makes the set more like reality every month and guards against ever repeating that mistake - the antidote to a stale set.
3 · As systems become agents that take actions and pursue multi-step goals, evaluation...
More autonomy means more to evaluate. Agents add tool-call accuracy, agent goal accuracy, and topic adherence, and errors can cascade across steps - so you increasingly evaluate the whole path, not just the destination.