Builder session 9 · Tracing & observability

Part 0

A score tells you what; a trace tells you why

Everything so far measures the outcome of a Recall request - was the retrieved chunk right, was the answer grounded, did the suite regress. None of it shows the path. When Recall returns a confidently wrong answer, the question is always the same: which step broke? Did the retriever miss, or did it find the right chunk and the LLM ignore it? You cannot answer that from a score. You answer it from a trace - a recorded timeline of every step in the request, with the exact inputs and outputs each one saw.

Live - presented in session Self-study - full depth after class ★ Try it now Sources covered at the end

★ What you build today A working mental model of trace / span / run, LangSmith tracing turned on over Recall with two env vars and one decorator, a map of the tracing-platform landscape (LangSmith, Langfuse, Phoenix, and where promptfoo fits), and the habit of attaching scores to individual spans so a wrong answer becomes an inspectable, per-step timeline.

Part 1 · covers trace, span, and run

Trace, span, run 8 min live

One user question to Recall is not one event - it fans out into a small tree of work. Retrieve the chunks. Call the LLM with those chunks. Parse the model's output into a clean answer. Each of those is a span: one unit of work, with its own inputs, outputs, latency, and token count. The whole tree, for that one request, is a trace. In LangSmith the word run means the same thing as a span - a single logged step.

🔍 Click to zoom - one Recall request as a trace of three spans

LiveSpan, trace, run - the three words3 min▶

The whole vocabulary of observability is three nested nouns. Get these and every tracing tool reads the same.

Span = one unit of work: an LLM call, a retrieval, a parse, a tool call. It records its own inputs, outputs, latency, and (for model calls) tokens. A span can have child spans.
Trace = the collection of all spans for one operation - one user request to Recall, top to bottom. It is a tree: a root span with children, laid out on a timeline.
Run = LangSmith's word for a span. When the LangSmith docs say "run", read "span". A run can be a child of another run, which is how the tree gets built.

LiveWhy observability matters for LLM apps2 min▶

Traditional software is mostly deterministic - same input, same output, and a stack trace when it breaks. LLM apps are neither. They are non-deterministic (the same prompt can give different answers) and multi-step (retrieve, generate, parse, maybe loop). When the final answer is bad, there is no stack trace pointing at the guilty line.

The failure is between steps. A wrong Recall answer might be a retrieval miss, a grounding failure, or a parser bug - three completely different fixes. Without a trace you are guessing which.
Observability turns the black box into a timeline. A trace shows the exact chunks the retriever returned and the exact prompt the LLM saw. You read down the tree and the broken step is obvious.
It is the substrate for everything else. Datasets, online eval (b10), and feedback all hang off traces. Turn tracing on first; the rest attaches to it.

Self-studyTurn tracing on with two env vars and one decorator4 min read▶

LangSmith is the fastest thing to wire up because it needs almost nothing from your code. Set two environment variables and it captures every LangChain call automatically. For your own plain functions, add the @traceable decorator and each call becomes a span in the tree.

bash · turn tracing onexport LANGSMITH_TRACING=true export LANGSMITH_API_KEY="ls-..." # from smith.langchain.com # optional: export LANGSMITH_PROJECT="recall-prod"

Python · @traceable makes each step a spanfrom langsmith import traceable @traceable(run_type="retriever") def retrieve(query: str) -> list[str]: return recall_retriever(query) # returns chunk ids, best first @traceable(run_type="llm") def generate(query: str, chunks: list[str]) -> str: return llm.invoke(build_prompt(query, chunks)) @traceable # the parent span def recall_answer(query: str) -> str: chunks = retrieve(query) # child span return generate(query, chunks) # child span recall_answer("how do I get a refund?") # one trace, three spans

The decorator builds the tree for you Because retrieve and generate are called inside recall_answer, LangSmith nests their spans under it automatically - you get the retriever span and the LLM span as children of the request, no manual parent-child wiring. Inputs and outputs are captured from the function arguments and return value.

Part 2 · covers the tools + attaching scores

The tools, and attaching scores to spans 8 min live

Tracing is a small standard with several implementations. LangSmith is hosted; Langfuse and Phoenix are open source and self-hostable. They all speak the same trace / span idea, and they all let you attach a score (also called feedback) to any span - which is the bridge back to everything you learned about judges. A score on the retriever span critiques retrieval; a score on the LLM span critiques generation. Same request, two independent verdicts.

🔍 Click to zoom - the tracing-platform landscape and where scores attach

LiveHosted vs open source3 min▶

All three do tracing well. The choice is about where the data lives and how much you want to run yourself.

LangSmith (hosted). A managed service by the LangChain team. Two env vars and @traceable and you are tracing. Tightest fit if you already use LangChain, and it has a built-in evaluate() runner (below).
Langfuse (open source, self-hostable). Same trace model, with an @observe decorator. Traces hold observations; you attach scores and build datasets. Pick it when data must stay on your own infrastructure.
Phoenix (open source, by Arize). Built on OpenTelemetry / OpenInference, so instrumentation is portable and vendor-neutral. Runs LLM, code, and human evals over the captured spans. Pick it when you want an open standard under the traces.

Real world A team ships Recall with LangSmith for a quick start, then a healthcare client requires all telemetry on-premises. Because both speak the same span model, they re-instrument with Phoenix on OpenTelemetry and keep every dashboard and score concept - only the backend moved.

LiveTracing platforms vs promptfoo2 min▶

promptfoo (from earlier sessions) is not a tracing platform, and the difference is worth naming so you reach for the right tool.

promptfoo is eval-first. It is a CLI that runs a fixed set of prompts against your golden cases and prints pass or fail. Perfect for a pre-deploy gate and CI. It does not give you a live-trace UI of production requests.
LangSmith / Langfuse / Phoenix are trace-first. They capture what actually happened in real (or replayed) requests, span by span, and let you inspect, score, and monitor over time.
You want both. promptfoo guards the merge (b8); a tracing platform watches production (b10). One is a gate, the other is a window.

Self-studyRun an eval over a dataset with LangSmith4 min read▶

Once tracing is on, LangSmith's evaluate() runs your system over a stored dataset and applies evaluators to each result - the same golden-set loop from b1, now hosted and traced. You create a dataset, write an evaluator function that returns a score, and hand both to client.evaluate() along with the target function.

Python · dataset + evaluator + client.evaluatefrom langsmith import Client client = Client() # 1. a dataset of golden examples dataset = client.create_dataset("recall-golden") client.create_examples( dataset_id=dataset.id, inputs=[{"query": "how do I get a refund?"}], outputs=[{"expected": "C-refund"}], ) # 2. an evaluator: returns a score for one result def hit_at_k(run, example) -> dict: got = run.outputs["chunks"][:3] ok = example.outputs["expected"] in got return {"key": "hit@3", "score": int(ok)} # 3. the target: your system as a function def target(inputs: dict) -> dict: return {"chunks": retrieve(inputs["query"])} # 4. run it - each example becomes a traced run with a score results = client.evaluate( target, data="recall-golden", evaluators=[hit_at_k], )

Attach user feedback to any child run Scores are not only for offline evaluators. In production you can attach a score to any child run - critique the retriever span and the LLM span with separate keys (say retrieval_ok and grounded) so a single bad answer produces two independent verdicts. That per-step feedback is exactly what b10's online eval samples and alerts on.

Tool	Model	Focus
LangSmith	Hosted · `@traceable`, `evaluate()`	Traces + hosted eval, tight LangChain fit
Langfuse	OSS, self-hostable · `@observe`	Traces/observations, scores, datasets on your infra
Phoenix	OSS · OpenTelemetry / OpenInference	Portable spans; LLM, code, and human evals
promptfoo	Eval-first CLI (no live-trace UI)	Pre-deploy gate over golden cases

Build-along · take it further

Sketch the spans of one Recall request ★ 10 min · pen and paper

Before you inspect a real trace, predict one. This is the skill: knowing what the tree should look like so a missing or wrong span jumps out.

Draw the tree. Take one Recall question and draw the root request span with its children: retriever span, LLM span, parser span. Nest them the way @traceable would.

Fill in each span. For every span write its inputs, outputs, latency, and (for the LLM) tokens. Be specific: the retriever's output is a list of chunk ids; the LLM's input is query plus those chunks.

Now the answer is wrong. Pick a failure. If the retriever span shows the wrong chunks, it is a retrieval bug. If it shows the right chunks but the LLM span's answer ignores them, it is a grounding bug. Which span would you open first, and what would you read?

Place the scores. Mark where a retrieval_ok score and a grounded score attach. Confirm they land on different spans - that separation is why tracing beats a single end-to-end pass/fail.

★ Recall observable after b9 Recall is now observable. Every request is a trace you can open, every step is a span with its own inputs and outputs, and scores attach per step so retrieval and generation are judged apart. A wrong answer is no longer a mystery - it is a timeline with one broken node. In b10 we point that instrument at live production traffic.

Homework

Before session b10 ◐ 45 min total

Set LANGSMITH_TRACING=true and LANGSMITH_API_KEY, add @traceable to your retrieve/generate/answer functions, run three questions, and open the traces. Confirm you see three spans nested under each request.
On one trace, attach two scores - a retrieval_ok on the retriever span and a grounded on the LLM span - and confirm they show up on different runs.
Build a small LangSmith dataset from your b1 golden set and run client.evaluate() with one evaluator. Read the per-example scores.
Read: LangSmith observability + evaluate() docs, the Langfuse tracing guide, and the Arize Phoenix quickstart. Note which one you would self-host and why.

Source material

Official sources covered

Taught from official docs. This page covers the core tracing and observability surface across four tools - the full metric-store and dashboard depth stays in each product's own guides.

✓LangSmith · observability + evaluate()Part 1-2 · env vars, @traceable, create_dataset, client.evaluate

✓Langfuse · tracing (OSS, self-hostable)Part 2 · @observe, traces/observations, scores, datasets

✓Arize Phoenix · OpenTelemetry / OpenInferencePart 2 · portable spans, LLM + code + human evals

◐promptfoo · eval-first CLIPart 2 · contrast only; gate vs window

Check yourself

Three questions before you go 🎯 ◐ 90 seconds

1 · In LangSmith terms, a "run" is...

In LangSmith, "run" is the word for a span: one unit of work (an LLM call, a retrieval, a parse) that can nest under a parent run to form the trace tree.

2 · The main reason observability matters more for LLM apps than for ordinary software is...

LLM apps give different outputs for the same input and chain several steps together. Without a per-step trace you cannot tell whether retrieval, generation, or parsing broke.

3 · Which statement is true of the tool landscape?

LangSmith is the hosted option, Langfuse and Phoenix are OSS and self-hostable (Phoenix on OpenTelemetry), and promptfoo is an eval-first CLI - a pre-deploy gate, not a live-trace window.

Builder session 9 cheat sheet · pin this

SpanOne unit of work - LLM call, retrieval, parse - with its own inputs, outputs, latency, tokens.

TraceThe tree of all spans for one request. Root span plus children on a timeline.

Run == spanLangSmith's word for a span. When docs say "run", read "span".

Turn it onLANGSMITH_TRACING=true + LANGSMITH_API_KEY, then @traceable on your functions.

Why it mattersLLM apps are non-deterministic and multi-step. A trace shows which step broke; a score cannot.

Hosted vs OSSLangSmith hosted; Langfuse (@observe) and Phoenix (OpenTelemetry) OSS and self-hostable.

vs promptfoopromptfoo is an eval-first CLI - a gate. Tracing platforms are the production window.

Scores on spansAttach feedback to any child run - critique retrieval and generation separately.