learn-ai-evals-with-phoebe / Leader session 3 of 6
Learn Evals with Phoebe · Leader track · Session 3 of 6

Offline vs online evaluation the gate and the smoke detector

Session 2 gave you the words for quality. This one gives you the two moments you measure it. There are exactly two kinds of evaluation, and they answer different questions: one runs before you ship, against a fixed set of questions with known-good answers, and decides whether you launch. The other runs forever after you ship, on live traffic, with no answer key, and tells you when the world has drifted out from under you. Teams that only do the first ship a product that is correct on launch day and slowly wrong ever after. You need both, and by the end of this hour you will know exactly why.

🟡 medium Leaders: C-level · managers · curious minds No code, ever 45 min
0-3 · Welcome 3-20 · Concepts 20-40 · Exercises 40-45 · Q&A
Part 0

Two moments, two very different jobs

Think of evaluation as two safety systems on a building. One is the inspection before anyone moves in - the gate that decides whether the building is fit to open. The other is the smoke detector that runs every hour after, catching the fire that starts long after the inspector left. Both are evaluation. Skip the inspection and you open something unsafe. Skip the smoke detector and you never find out when it goes wrong in use. AI systems need both, and most teams have only installed the inspection.

Live - presented in session Self-study - read after class ★ Try it now prompt Sources covered at the end
★ What you walk out with today The clean split between offline eval (before ship, fixed answer key, gates the launch and catches regressions) and online eval (after ship, live traffic, no answer key, catches drift), why neither replaces the other, how user thumbs up/down feeds the online signal, and five questions to put to your data team this week.
Part 1 · covers the two kinds of eval

Two kinds of eval 9 min live

Evaluation happens at two moments in a system's life. Offline eval runs before launch and again on every change, against a fixed set of questions with reference answers. Online eval runs continuously after launch, on real production traffic, with no reference answer to compare against. Same discipline, two entirely different setups.

OFFLINE · before ship fixed dataset with reference answers = the launch gate OFFLINE · every change re-run same dataset on each tweak = catches regressions ONLINE · after ship sampled live traffic, no reference answer = catches drift SHIP repeat on every version Offline gates the launch and guards every change. Online watches the world after ship.
🔍 Click to zoom - the eval lifecycle: offline before and on every change, online forever after
LiveOffline eval - the launch gate you build once and re-run forever3 min

Offline eval runs before you ship. You assemble a fixed set of representative questions, each paired with a known-good reference answer, and you grade the system against that answer key. The score is the gate: high enough, you launch; not high enough, you do not.

  • The answer key is the whole point. Because you know the right answer for each question, grading is clean and repeatable. The same test run today and next month is directly comparable.
  • It re-runs on every change. Swap a model, tweak a prompt, change how documents are fetched - re-run the exact same set. If the score drops, the change broke something. That is a regression, caught before a user ever sees it.
  • Its blind spot: it only ever asks the questions you thought to put in the set. It cannot see the questions real users will actually ask tomorrow.
LiveOnline eval - the smoke detector on live traffic3 min

Online eval runs after you ship, continuously, on the real questions real users are asking right now. The defining difference: there is no answer key. Nobody wrote a reference answer for a question that was asked thirty seconds ago, so online eval grades differently - it checks the answer against the material it used, watches for signals like low groundedness, and folds in what users tell you.

  • No reference answers, ever. Live questions arrive without a known-good answer. So online grading leans on reference-free checks and on user feedback rather than an answer key.
  • You sample, you do not grade everything. Grading every single production answer costs real money, so teams grade a slice - a sampling rate. A rate of 0.1 means one in ten answers gets evaluated: enough to spot a trend, cheap enough to run always.
  • You can aim the sampling. Teams often focus grading on the answers users flagged as bad - a thumbs-down - so the budget goes where the trouble is.
LiveSide by side - the same discipline, two setups3 min

The cleanest way to hold the difference is a table. Read it across each row and the split is obvious.

Offline evalOnline eval
WhenBefore ship, and on every changeContinuously, after ship
On what dataFixed set of chosen questionsReal, live production traffic
How gradedAgainst known reference answersNo reference - reference-free checks + user feedback
Catches whatRegressions between versionsLive-quality issues and drift
Part 2 · covers why one is never enough

Why you need both 8 min live

Offline is the gate before launch; online is the smoke detector after. Neither does the other's job. Offline cannot see the real world coming, and online cannot gate a launch that has not happened. A team with only one has a blind spot exactly where the other would have been looking.

OFFLINE the gate good enough to open? SHIP 🚀 ONLINE · smoke detector watches live traffic, catches the fire that starts after launch The world drifts: documents change, users ask new things, the model updates under you. Only online sees it. Offline can't see the real world. Online can't gate a launch. You need both.
🔍 Click to zoom - the gate and the smoke detector, each covering the other's blind spot
LiveWhy offline alone leaves you blind3 min

Offline eval is frozen in time. Your test set was written on a particular day, reflecting the questions and the world of that day. But the world does not hold still after you ship, and offline eval never notices.

  • The documents drift. A policy changes, a price updates, a product is discontinued - and the system keeps confidently quoting the old material. Your offline set, with its old answers, still passes.
  • Users ask new things. Real traffic is full of questions nobody put in the test set. Offline eval literally cannot grade a question it does not contain.
  • The model changes under you. When a provider updates the underlying model, behavior can shift without a single line of your own changing. Offline catches this only if you happen to re-run - online catches it live.
LiveWhy online alone cannot gate a launch3 min

The reverse is just as true. Online eval is powerful, but it only exists once you are live - which is too late to decide whether you should have gone live.

  • No answer key yet. Before launch there is no production traffic and no way to know the right answers at scale. Online eval has nothing to grade until real users arrive.
  • You cannot regression-test a change safely. Deciding whether a new prompt is better belongs in a controlled offline run against known answers - not by shipping it to customers and watching complaints.
  • Feedback is a signal, not a gate. User thumbs up and down are gold for online monitoring, but you cannot ask users to grade a product that has not shipped.
Real world

The bot that aged badly. A team (anonymized) shipped a support bot that passed its offline eval cleanly - every reference answer matched. Three months later a refund policy changed. The offline set still had the old questions and old answers, so it kept passing. Meanwhile the live bot confidently quoted the retired rule to real customers for weeks. The only thing that could have caught it was online eval watching live traffic - and the team had not installed one. The first signal was a spike in complaints.

LiveUser feedback - your cheapest online signal3 min

The simplest online eval is one you already have the ingredients for: the thumbs up and thumbs down next to an answer. Every user who reacts is grading your system for free, in the real world, on a real question.

  • It becomes a number. A thumbs up counts as 1, a thumbs down as 0. Averaged over traffic, that is a live quality signal you can chart and watch for dips - and it can be attached to any step of the process, not just the final answer.
  • It aims the expensive grading. Rather than sampling answers at random, focus the deeper evaluation on the ones users flagged as bad. The complaints tell you where to look.
The leader move If your product has no way for a user to say "that answer was wrong", you have switched off your cheapest smoke detector. Ask whether feedback is captured, and whether anyone is watching the trend - not just the individual thumbs.
Exercise 1 of 2

Offline catches this, online catches that ★ 12 min · pen and paper

Pick one AI use case you own or care about. For that case, sort what each kind of eval would catch - and, crucially, what only online would ever see.

Name the use case in one line, and the fixed set of questions you would put in its offline test - the handful you are sure it must get right.

List two things offline eval would catch for this case - for example, a prompt change that quietly made answers worse, or a model swap that dropped quality.

List two things only online eval would catch - for example, a source document changing after launch, or users asking a whole category of question you never anticipated.

Ask the hard one: if the most likely failure for this case is drift after launch, and you only have offline eval, how long would you stay blind? That gap is the case for building online.

★ Your eval-planning partner (paste into any chat AI)I am planning evaluation for an AI use case. I will describe it. Sort the risks into two lists: (1) things offline evaluation would catch - failures visible when testing against a fixed set of questions with known answers, before or during changes; and (2) things only online evaluation would catch - failures that appear only on live production traffic after launch, like drift, new question types, or a model update. Then tell me, for this case, which of the two I most need and why. My use case is: [describe it]
Exercise 2 of 2

Design your online signal ★ 8 min · your next AI review

Online eval only works if you decide, in advance, what to watch and how much of it. Sketch the online signal for your use case in four decisions.

What would you sample? You will not grade every answer - it costs too much. Decide the slice: a random one in ten, or every answer on a high-stakes topic? Name your sampling rate in plain terms.

What feedback would you capture? Thumbs up/down is the minimum. Would you also capture a reason, a follow-up "did this solve it?", or an escalation-to-human signal? Each is a live grade.

What would trigger a deeper look? Decide the rule: "grade every thumbs-down answer", or "when the weekly thumbs-up average drops below X, investigate". Aim the expensive grading at the trouble.

Who watches the trend, and how often? A signal nobody looks at is not a smoke detector, it is a disconnected one. Name the owner and the cadence.

Sampling is a budget dial, not a corner cut A one-in-ten sampling rate is not laziness - it is how you afford to watch quality every single day instead of only when someone remembers. The trend on a sample tells you as much as grading everything, at a tenth of the cost.
Homework

Before session a4 ◐ 30 min total

★ Questions to ask your data team this week
  1. Do we have an offline eval - a fixed set of questions with known answers we re-run before every change?
  2. Do we have any online eval on live traffic, and if so, what sampling rate do we grade at?
  3. If a source document changed after launch and the bot kept quoting the old one, how would we find out - and how fast?
  4. Do we capture user feedback like thumbs up/down, and does anyone watch the trend rather than just individual complaints?
  5. How long, honestly, would we stay blind to a quality drop in production - and what would shorten it?
Source material

Official sources covered

The leader track teaches from published engineering guidance and course curricula - no vendor marketing. This page covers:

LangSmith · Evaluation concepts & Online evaluationsPart 1-2 · offline against a dataset with reference outputs · online on production traces, sampled, no reference · feedback scores
Anthropic · Develop your testsPart 1 · build empirical evals before you trust the system · the launch gate
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The defining difference between offline and online evaluation is...

Offline runs before ship against a fixed dataset with reference answers and catches regressions. Online runs continuously on live traffic where no reference answer exists, using reference-free checks and user feedback.

2 · A support bot passes its offline eval every week but starts giving wrong answers in production after a policy changes. What happened?

Offline eval is frozen to the day it was written. When documents change under a live system, the old test set keeps passing. Only online eval, watching real traffic, sees the drift.

3 · A sampling rate of 0.1 on online eval means...

Grading every live answer is expensive, so teams grade a slice. A rate of 0.1 grades one in ten - the sampling rate is the budget dial that lets you watch quality every day. Grading can be aimed at feedback-flagged runs.

Leader session 3 cheat sheet · pin this

Two kindsOffline = before ship, fixed answer key. Online = after ship, live traffic, no answer key.
Offline = the gateGrades against known reference answers. Decides launch, and re-runs on every change.
Catches regressionsRe-run the same set after a tweak. Score drops = the change quietly broke something.
Online = smoke detectorRuns forever on production traffic. No reference answers - uses reference-free checks + feedback.
The world driftsDocuments change, users ask new things, the model updates under you. Only online sees it.
Sampling rateGrade a slice, not everything. 0.1 = one in ten. The budget dial for continuous watching.
User feedbackThumbs up = 1, down = 0. A free live signal, attachable to any step. Aim grading at the flags.
You need bothOffline can't see the real world. Online can't gate a launch. Trust needs the pair.