What this course is, and what it is not
This is not a course about eval metrics, and it is not a course about platform architecture. Three siblings own those: Learn Evals owns retrieval and generation metrics, Agent Launch owns the go/no-go decision, and AI Infrastructure owns the platform layers underneath. This course owns one thing they all assume and none of them teach: the trace.
Six words, and what each one is for 7 min live
Tracing has a small vocabulary and it is worth being pedantic about, because almost every argument about observability tooling turns out to be two people using one of these words differently.
LiveWhy a log line is not enough4 min▶
Most teams arrive at AI observability already logging. The logs look adequate right up until the first real incident, and then they fail in a specific way worth naming.
| Log lines | A parented trace | |
|---|---|---|
| Shape | A flat sequence, ordered by clock | A tree, ordered by causation |
| Concurrency | Interleaved and ambiguous - two parallel tool calls become four confusing lines | Two sibling spans with overlapping intervals. Unambiguous |
| "Where did the time go?" | Subtract timestamps by hand, hope nothing is missing | Read it off the waterfall |
| Nesting | Inferred from indentation, if someone bothered | Explicit parent id |
| Under an agent loop | Collapses. Iteration 3 looks exactly like iteration 1 | Each iteration is its own subtree |
The honest summary: logs answer "what did the code say", traces answer "what did the system do". For a single-shot model call the gap is narrow enough that logs get you by. For anything with a loop in it, the gap is the whole problem.
Self-studyThe three questions a trace must answer to be worth storing3 min read▶
Instrumentation is not free, so it is worth having a test for whether a span is earning its keep. A trace is worth storing when it can answer all three of these about a single request:
- Where did the time go? Every second of wall clock should sit inside some span. If the child spans do not add up to the parent, the missing time is exactly where your next incident will hide.
- What did each step actually get and return? Not the full text necessarily - the counts, the ids, the row counts, the document counts. "Retrieval ran" is nearly useless; "retrieval returned zero documents" is the whole diagnosis.
- Did anything fail, including quietly? A caught-and-handled failure is still a failure and still belongs in the record. This is the one teams get wrong most often, and it is why a broken system can hold a green dashboard for a week.
If a span cannot contribute to any of those three answers, it is costing you storage and cardinality for nothing. Session b7 puts numbers on that trade.
What happens when one call becomes an agent 6 min live
A single model call is a trivially easy thing to observe: one span, two token counts, done. Everything hard about this course arrives when the call gains the ability to decide what to do next.
Instrument one call, by hand, once ★ 12 min · everyone builds
Do this manually exactly once, so that every auto-instrumentation library you use afterwards is something you understand rather than something you trust. Any language with an OpenTelemetry SDK works; the shape below is Python.
Wrap your simplest model call in a span. Name it for the operation, not the function: chat <model> rather than call_llm_v2. Span names become dashboard rows, and a dashboard row named after a function is a dashboard row nobody reads.
Set the attributes the standard already has names for - model, token counts, finish reason. Session b2 goes through the exact current keys. Use the standard keys even if you are only ever going to look at them yourself: the day you switch tooling, conventional attributes port and bespoke ones do not.
Now deliberately break it. Raise an exception inside the span, and confirm the span comes back with error status. Then catch the exception, handle it, and watch the span go green while the failure is still real. Sit with that for a second - you have just reproduced the exact mechanism behind the incident this course is built around.
Add a second span around the whole request, as the parent. Confirm the child's duration sits inside the parent's, and subtract: the difference is your first piece of unaccounted time. Write the number down.
with tracer.start_as_current_span(f"chat {model}") as span: · span.set_attribute("gen_ai.request.model", model) · span.set_attribute("gen_ai.usage.input_tokens", resp.usage.input_tokens) · and on the failure path, span.set_status(Status(StatusCode.ERROR)) plus span.record_exception(e) - even when you handle the error and carry on.
try / except around anything an AI request depends on, set the span status in the except block before you return your fallback. That single reflex is the difference between the incident in this course taking twenty minutes to find and taking a week.
Read the trace you will be living in ★ 8 min · explore
This is the incident the builder track hunts. Open it now with everything switched off, the way it actually shipped, and just look at the gap. Session b5 does the full hunt; today you only need to feel how innocent a badly instrumented trace looks.
Before touching a lever, read the red bar under the waterfall. Two thirds of the run is inside no span at all - and nothing on screen looks wrong. This is what your on-call engineer saw.
Switch on Tool spans alone. Notice how little it helps. Tools are the easy thing to instrument and they were never where the time went - a very common way to feel productive without getting closer to the bug.
Now switch on Retriever spans. The gap closes to zero and a 4.2 second step appears out of nowhere. You have not changed the system at all; you have changed only what it reports about itself.
Leave it there for today. The remaining levers, and the five questions a complete diagnosis has to answer, are session b5.
Try it yourself - this week ◐ 40-50 min total
- Take one real AI request in your own system and account for its wall clock. Add up every span you currently emit and subtract from the total. The unaccounted percentage is the single most useful number you will produce this week, and for most teams it is above fifty.
- Find every
try / exceptin the path of an AI request and count how many set span status on the failure path. If the answer is zero, you cannot currently distinguish "worked" from "failed quietly", and neither can your dashboard. - Write down the three questions from Part 1 and check them against a trace you already store. Any question you cannot answer is a coverage gap with a name, which makes it a ticket rather than a vague unease.
- Draw your own version of the Part 2 tree for one of your agents, from memory, then compare it against a real trace. The difference between the tree you think you have and the tree you actually emit is worth seeing before an incident shows it to you.
Official sources covered
Every convention and claim on this page traces to a primary source - the OpenTelemetry specification and its GenAI semantic conventions - catalogued in this course's source map. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your child spans add up to 2.2 seconds and the parent span says 6.8 seconds. What have you learned?
Unaccounted time is the most actionable number in a trace. The parent is measuring exactly right - it is measuring work you have chosen not to describe. Every serious AI incident in this course lives in that gap, because a step you do not instrument cannot appear slow, cannot appear empty and cannot appear failed.
2 · Which part of an agent run does vendor SDK auto-instrumentation reliably give you, and why is that awkward?
Auto-instrumentation wraps what the vendor ships, and the vendor ships the model client. That is genuinely useful and it is also the layer that fails least often. The failures that reach your users tend to sit in the glue - an empty retrieval, a stale cache, a tool called with the wrong argument - and none of that appears unless someone decided it should.
3 · You catch an exception, fall back to cached data, and return a successful response. What must happen to the span?
This is the mechanism behind the incident the whole course chases. The user got a response, so nothing looks wrong, and the fallback quietly served nineteen-hour-old data. If the caught exception never becomes a span status or a recorded event, your monitoring is structurally incapable of telling you the difference between working and degraded - and it will hold a green dashboard through an entire incident.