learn-ai-evals-with-phoebe / Builder session 10 of 10
Learn Evals with Phoebe · Builder track · Session 10 of 10

Online eval and drift: keeping Recall honest in production

This is the last builder session, and it closes the loop. Everything so far ran offline - golden sets, judges, suites, traces - all before the deploy. But production is where the truth lives, and production has no reference answers. Here you learn to evaluate live traffic: sample a slice of real requests, judge them without a ground truth, fold in user thumbs, and watch for drift - both the slow shift in what users ask and the quieter shift in what the right answer even is. By the end Recall is not just measured once; it is continuously measured, and you have a monitoring design you could ship. Then Recall graduates.

🔴 Builder track · hardest Practitioners · Python Online eval · drift detection Final session · 10 of 10
0-3 · Welcome 3-18 · Concepts 18-40 · Build-along 40-45 · Close
Part 0

Offline proves it works once; online proves it keeps working

Every eval before this one had an answer key. The golden set knew the right chunk; the judge had a rubric; the suite had a baseline. Production has none of that. Real users ask questions you never wrote down, and nobody labels the right answer in real time. So online evaluation trades the answer key for two other signals: reference-free judging on a sample of traffic, and the users themselves - a thumbs up or down. On top of that, you watch for drift, because a system that passed every test in June can quietly rot by September without a single line of code changing.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end
★ What you build today A production monitoring design for Recall: a sampling strategy that controls cost, reference-free online eval on live traces, user feedback captured as scores, and two drift signals - data drift and concept drift - with an alert threshold. The final piece that turns Recall from "shipped" into "continuously trustworthy".
Part 1 · covers online evaluation

Online evaluation 8 min live

Online eval runs continuously on production traces, with no reference answers. You cannot judge every request - that would double your LLM bill - so you sample: a rate like 0.1 means score 10% of traffic. On that sample you run reference-free heuristics and LLM-judges, fold in any user feedback, and push the result to a dashboard with alerts. Because judging costs money, a smart pattern is to judge everything cheaply and reserve the expensive LLM-judge for runs the users already flagged.

Production every live request as a trace Sample rate 0.1 = 10% of traffic Judge (no ref) LLM-judge + heuristics + feedback Dashboard + alerts on score drop User feedback thumbs up=1 / down=0 No answer key in prod. Sample to control cost, judge reference-free, and let users vote with thumbs.
🔍 Click to zoom - the online eval pipeline: sample, judge, dashboard, alert
LiveWhat changes when you go online3 min▶

Online eval is the offline loop with the answer key removed and a cost dial added.

  • No reference answers. You cannot compute Hit Rate or exact-match on live traffic - nobody labeled it. You lean on reference-free judges (is the answer grounded in the retrieved chunks? is it relevant?) and on the users.
  • Sampling controls cost. A sampling rate like 0.1 scores 10% of requests. Higher catches issues faster; lower saves money. It is a dial you set against your budget, not a fixed number.
  • User feedback is a first-class score. A thumbs up is a 1, a thumbs down is a 0, attached to that request's run. Cheap, honest, and it needs no judge at all.
  • Filter the expensive judge. Run cheap heuristics on the whole sample, but reserve the pricey LLM-judge for runs users flagged with a thumbs-down - you spend judgment where the signal already points.
Self-studyAttach feedback and sample the online eval4 min read▶

User feedback is a numeric score attached to a run by its id - the same run/span from b9. A thumbs up sends 1, a thumbs down sends 0. Then you decide which sampled runs get the expensive judge.

Python · capture a thumb + sample the judgeimport random from langsmith import Client client = Client() # user clicked thumbs up/down on a Recall answer def on_user_vote(run_id: str, thumbs_up: bool): client.create_feedback( run_id, # the run/span from b9's trace key="user_thumb", score=1 if thumbs_up else 0, # up=1, down=0 ) # online eval: judge a sample, and always judge flagged runs SAMPLE_RATE = 0.1 # 10% of traffic def maybe_judge(run): flagged = run.feedback.get("user_thumb") == 0 if flagged or random.random() < SAMPLE_RATE: score = grounded_judge(run) # reference-free LLM-judge client.create_feedback(run.id, key="grounded", score=score)
Sampled, not skipped Sampling is not laziness - at scale, judging every request with an LLM can cost more than serving them. A 10% sample of a busy endpoint is thousands of judged calls a day: plenty to see a trend and fire an alert, at a tenth of the bill. Push the rate up temporarily when you suspect trouble, back down when it is quiet.
Part 2 · covers drift + the graduation

Drift, and the graduation 8 min live

A monitored system can still degrade without any code change, and there are two flavors. Data drift is when the inputs shift - users start asking about things they never used to. Concept drift is sneakier: the inputs look the same, but the right answer has changed underneath you. Layer 1 detects that something moved; layer 2 sends a sample of the drifted prompts to an LLM-judge to classify why - a new topic, an intent shift, more complexity, a change in language style.

Data drift input distribution shifts detect: input-embedding shift alert on threshold Concept drift inputs look similar but the right answer has changed Layer 2 LLM-judge on a sample of drifted prompts Classify cause new topic · intent shift · complexity · style Layer 1 says something moved; layer 2 says what kind. That is the difference between an alarm and a diagnosis.
🔍 Click to zoom - data drift vs concept drift, and the layer-2 cause classifier
LiveWhat drifts in a RAG system3 min▶

Recall has three moving parts that can each drift the ground under it, even while your code sits still.

  • Corpus staleness. The refund policy changes but the indexed chunk still says the old thing. Inputs look normal, retrieval looks fine, and the answer is confidently out of date - classic concept drift.
  • Model updates. The provider ships a new model version and your grounded, well-behaved prompt suddenly phrases things differently or refuses more. Same input, different behavior.
  • User behavior. A product launch sends a wave of questions about a feature Recall has never seen - a clean example of data drift, a shift in the input distribution.
Real world Recall runs quietly for months. Then support updates the refund window from 14 to 30 days but nobody re-indexes the doc. No input changed, no code changed, yet every refund answer is now wrong. Data-drift monitors stay green; only a concept-drift check - or a spike in thumbs-down on refund questions - catches it.
LiveDetecting drift without a freshness rule2 min▶

Detection is mostly a statistics-plus-threshold job, with an honest gap where a magic number would be.

  • Data drift: watch the input distribution. Embed incoming questions and compare this window's distribution to a baseline. For high-dimensional embeddings, Wasserstein distance is a common measure; alert when it crosses a threshold you tune.
  • Concept drift: watch the outcomes. Inputs can look identical while quality falls - a drop in the grounded score or a rise in thumbs-down on a topic is your signal that the right answer moved.
  • No universal freshness cadence. There is no authoritative "re-index every N days". How often to refresh the corpus is a design decision you make from how fast your source-of-truth changes - present it as a trade-off, not a constant.
Two layers, two jobs Keep detection and diagnosis separate. Layer 1 is cheap and always-on: an embedding-distribution monitor with a threshold that just says "something shifted". Layer 2 is expensive and on-demand: sample the drifted prompts and let an LLM-judge classify the cause - new topic, intent shift, more complexity, changed language style - so a human gets a diagnosis, not just an alarm.
Self-studyThe graduation - Recall, from vibe to continuously measured3 min read▶

This is the last session, so step back and see the whole ladder you climbed over the builder track. Each rung is a session, and each one made Recall a little harder to fool.

  • b1-b3 · the offline foundation. A golden set, trustworthy questions, and the metric suite - Recall got its first real number.
  • b4-b7 · judging the ungradeable. LLM-as-judge, groundedness and faithfulness, rubrics and pairwise - Recall's answers, not just its retrieval, got measured.
  • b8-b10 · making it durable. A regression suite as a merge gate, tracing so failures are inspectable, and now online eval and drift so the measurement never stops.

Recall began as a RAG assistant in the RAG course. It is now grounded, gated, observable, and continuously measured against live traffic. That is a system you can actually trust in front of users - and trust you can defend with numbers.

The next frontier: agent eval Recall answers in one shot. The moment you let it plan, call tools, and loop - an agent - a new evaluation surface opens: was the trajectory right, not just the final answer? Did it call the right tool with the right arguments? That is where orchestration and evaluation meet, and it is exactly what the LangChain course picks up next.
Build-along · take it further

Design Recall's production monitoring ★ 12 min · pen and paper

The final exercise is a design, not a script. Write the monitoring plan you would actually ship for Recall - four decisions, each defended in a sentence.

What to sample. Pick a sampling rate and justify it against a rough traffic and cost estimate. Would you always judge thumbs-down runs on top of the sample? Say why.

What feedback. Decide what the user can send - thumbs up/down at minimum - and which run it attaches to. Note how a thumbs-down would trigger the expensive judge from Part 1.

What drift signal. Choose one data-drift signal (embedding-distribution shift on incoming questions) and one concept-drift signal (a fall in grounded score or a rise in thumbs-down on a topic). State what baseline you compare against.

What alert threshold. Set a threshold that pages a human, and be honest that it is a starting guess you will tune. Decide how re-indexing cadence gets decided - remember there is no universal freshness number.

★ Recall graduates That is the builder track. Recall started as a RAG assistant, and across ten sessions it became something you can defend: grounded in its sources, gated on every change, observable step by step, and now continuously measured against live traffic with drift alarms watching its back. Evaluation is no longer a thing you do before launch - it is a thing that never stops. Recall is built, and it is honest. Congratulations - go keep something in production honest.
Homework

After the track ◐ light · 30 min

Source material

Official sources covered

Taught from official docs. This page covers the core online-eval and drift surface. One honest gap: there is no single authoritative re-index cadence, so freshness is presented as a design decision, not a number.

✓LangSmith · online evaluationsPart 1 · continuous, reference-free, sampled eval on production traces
✓LangSmith · attach user feedbackPart 1 · create_feedback, thumbs up=1 / down=0 on any run
✓AWS · data drift vs concept drift guidancePart 2 · input-distribution shift vs changed input-to-output relationship
◐Re-index / freshness cadencePart 2 · no single number; presented as a design trade-off
Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · The defining feature of online evaluation versus offline is...

Online eval has no answer key - it judges live traffic reference-free, and samples (e.g. 10%) to control cost while still seeing trends.

2 · The refund policy changed but the indexed chunk was never updated, so answers are now wrong while inputs look normal. This is...

Concept drift is when the input-to-desired-output relationship changes: the questions look the same, but the correct answer moved. Data drift is a shift in the inputs themselves.

3 · A thumbs-down from a user is best captured as...

User feedback is a numeric score (up=1, down=0) attached via create_feedback to any run. Flagged runs are exactly where you spend the pricier LLM-judge.

Builder session 10 cheat sheet · pin this

Online evalContinuous, on production traces, no reference answers. The offline loop minus the answer key.
SamplingRate like 0.1 scores 10% of traffic. A cost dial - turn it up when you suspect trouble.
User feedbackNumeric score via create_feedback: thumbs up=1, down=0, attached to any run.
Filter the judgeCheap heuristics on the whole sample; reserve the LLM-judge for flagged runs.
Data driftInput distribution shifts. Detect via input-embedding shift (Wasserstein) + threshold alert.
Concept driftInputs look similar but the right answer changed. Watch outcomes, not just inputs.
Two layersLayer 1 says something moved; layer 2 LLM-judge classifies the cause.
Freshness = designNo universal re-index cadence. Decide it from how fast your source of truth changes.