A diagnosis is not a feeling, it is five answers
Tuesday 09:14. Beacon's Insight Bot is sending executives stale revenue numbers, intermittently. No error is raised, no alert fires, the p95 latency dashboard is green, and the answers read as confidently as ever. An engineer opens the trace at 09:20 and closes it at 09:21, because there is genuinely nothing in it to see. Today you find the bug, and the thing you change to find it is not the application - it is what the application reports about itself.
gen_ai.retrieval.cache_fallback and cache.entry_age_hours, are application-defined rather than conventions, because the registry has no name for either - which is itself worth noticing.
What you have to be able to answer 6 min live
"The bot gave a wrong number" is a report, not a diagnosis. A diagnosis is five specific answers, and the useful thing about writing them down is that each one has a different coverage prerequisite. You cannot buy them in bulk and you cannot get them from the same lever.
The order matters more than it looks. The first question - which step is slow - is the one every team instruments for, because latency is what dashboards are made of. It is also the least useful of the five here: the retrieval step in this incident is 4.2 seconds of a 6.8 second run, and knowing that tells you where to look and nothing about what is wrong. Slow was never the bug. Slow was the symptom that made the bug findable.
1. Does the wall clock add up? root duration vs the union of the children 2. What is the widest span? and is it a model call or something else 3. What did that step return? counts, row counts, document counts, ids 4. Did anything fail quietly? status, error.type, recorded exceptions 5. Is this run unusual? thread by conversation id and compare If step 1 fails, stop. Everything after it is a guess about a system you are not currently observing.
Five links, and the last one is silent 5 min live
Here is the whole incident, resolved, so you can watch it appear rather than guess at it. Read the right-hand column as a shopping list: each link in the chain has exactly one piece of telemetry that would have named it, and the team had none of them.
The climb: eight rungs from blind to provable ★ 12 min · watch, then argue
Step through these in order and read the caption before you look at the meters. The application code never changes across these eight rungs. Only what it reports about itself changes, which is the entire claim of this course made falsifiable in a browser.
Rung 2 adds tool spans and takes visible wall clock from 32.4% to 38.2%. The diagnosis stays at 0 of 5. Sit with that for a moment, because it is the most expensive lesson on the page: tools are the easy thing to instrument, they are what a framework integration hands you first, and in this incident they were never where the time went. A team can spend a sprint on tool instrumentation, feel productive, ship a nicer waterfall, and be exactly as blind as they started.
Rung 3 adds the retriever span and the number moves to 100%. The 4.2 second step appears out of nowhere and the unaccounted-time warning disappears entirely. The diagnosis goes to 1 of 5. This is worth naming precisely: you now have complete coverage of time and 20% of a diagnosis. Accounting for every millisecond tells you where to look. It does not tell you what you are looking at.
Rung 4 adds the retrieval attributes and the sentence changes shape. gen_ai.retrieval.documents.count is zero and gen_ai.retrieval.cache_fallback is true, so the step is not slow, it is empty - and something quietly substituted cached data for the answer. 2 of 5. Rung 5 goes one level deeper into the database and cache children and locates the root cause physically: the query returned zero rows after 4.1 seconds, and a cache entry 19.4 hours old was served in its place. That is the stale number the CFO read at 09:14, now pinned to an exact span. 3 of 5.
Rung 6 is the one that explains the silence. The query did not return empty, it timed out - and someone caught the exception, fell back to cache and returned success. That swallowed exception is why nothing alerted and why the dashboard stayed green through the entire incident. 4 of 5. Rung 7 threads conversations and adds the token attributes: the failure is intermittent rather than constant, which is a different fix, and the run's cost becomes knowable at $0.0180 using the per-span token work from session b4. 5 of 5. You can write the postmortem from the trace.
try: rows = warehouse.query(sql, timeout=4.0) except QueryTimeout as e: span.set_status(Status(StatusCode.ERROR)) # <- the missing line span.record_exception(e) # <- the other missing line span.set_attribute("gen_ai.retrieval.cache_fallback", True) rows = cache.get("metrics_snapshot") # served 19.4 hours stale
Free play: get to 5 of 5 yourself ★ 10 min · everyone builds
Same incident, all eight levers unlocked, nothing scripted. Work it yourself, and pay more attention to the levers you did not need than to the ones you did.
Start from nothing and get the diagnosis to 5 of 5. Then write down your minimum lever set - the smallest collection of switches that answers all five questions. Compare it with the person next to you; most people arrive with at least one lever they did not need.
The minimum set is five levers: retriever spans, retrieval attributes, database and cache children, error propagation, and session IDs. Tool spans and token attributes are not required for any of the five questions - tools buy you sequencing, tokens buy you the cost figure - and content capture buys nothing at all here. Knowing which of your levers are load-bearing is what turns an instrumentation backlog into a priority order.
Now switch on tool spans alone and watch the span count grow while the diagnosis stays at 0 of 5. Then switch on retriever spans alone, with tools off, and watch the diagnosis reach 1 of 5 immediately. One lever is instrumentation theatre on this incident and the other is the whole game, and nothing about their names tells you which is which in advance.
Turn on the database and cache children and watch the visible wall clock stay exactly where it was. Those spans are children of the retrieval span, so they add no new accounted time - the time was already counted by their parent. Accounting for time is a property of the top level of the tree; depth beyond that buys meaning, not minutes. This is the single most common misreading of a coverage metric.
Finally, turn the token attributes on and off and watch the cost meter flip between $0.0180 and unknown. Nothing about the run changed. The money was always being spent; without the attributes it was simply not attributable to this user, this feature or this failure.
What the anti-lever actually costs, at every scale ★ 6 min · read the ratio
Rung 8 added no answers. Here is what it added instead, computed live across five traffic volumes from a modelled 4 KB base trace, 38 KB of message content and $0.28 per GB-month. Substitute your own storage price and your own traffic; the interesting column is not the dollars.
Read the Multiple column top to bottom. It is 10.5x at every single row, because it is a ratio of bytes and ratios do not care about scale. Content capture does not cost a little more when you are small and a lot more when you are large - it costs the same order of magnitude more at every scale you will ever operate at.
Now read the dollars at 1,000 runs per day, where the extra rounds to $0. That row is how content capture gets switched on during a quiet week in a pre-production environment and never switched off. The decision was made when it was free and it is still in force at a million runs a day.
Then read the dollars at the top row and notice they are still modest. Money is not the argument - the argument is that you have converted a telemetry store into a system holding raw customer text, usually in third-party SaaS, usually with far broader internal read access than the production database the same text came from. Session b10 does the legal and retention arithmetic properly.
Decide your own policy in one sentence before you leave: metadata at 100%, content sampled, redacted, or scoped to the traces that need it. Write down which of the three you chose. The specification's own default is the first pattern - do not record instructions, inputs or outputs - and the second is described as best suited to pre-production.
Try it yourself - this week ◐ 50-60 min total
- Take the five questions to your own most recent incident and answer them from the telemetry you already had at the time - not from what you eventually worked out in Slack. Score yourself out of five. The gap between that score and five is your instrumentation roadmap, in priority order, with no strategy work required.
- Compute visible wall clock on one real trace: root duration minus the union of your top-level children, over root duration. If you are below 60% you are in rung 1 or 2 territory, and the useful next move is almost certainly one span around whatever data-fetching step your team thinks of as plumbing.
- Grep every
exceptblock in the path of an AI request and count how many set span status before returning a fallback. Each one that does not is a private, silent rung 6 waiting to happen. Fix the one wrapped around your data source first. - Find out whether content capture is currently on in any environment you own, and who turned it on. If the answer is "it was on by default in the library we adopted", check the retention window and the access list on that store today rather than at the next audit.
- Run the free-play board once more and deliberately try to reach 5 of 5 with the fewest levers. Then map each of those five levers to a named owner and a ticket in your own system. Five tickets is a completely tractable quarter, and it is the difference between this session being interesting and this session being useful.
Official sources covered
The incident is constructed; the attribute keys and the arithmetic are not. Every key on the simulated trace comes from a primary source - the OpenTelemetry GenAI semantic conventions and the general database conventions - catalogued in this course's source map and verified 2026-08-13. This page covers:
error.type and recorded exceptions as the only signal a caught failure ever leavesDemo 1 rung 6 · the mechanism behind the green dashboardgen_ai.retrieval.documents.count, and db.response.returned_rows from the database conventionsPart 2 and Demo 1 rungs 4-5 · empty is not slowgen_ai.conversation.id as the join key across turns, and the spec's prohibition on fabricating oneRung 7 · intermittent versus constant is a different bugcache_fallback, entry_age_hours)Constructed for teaching · no convention exists for either key, and the registry is worth checking before you invent your ownThree questions before you go 🎯 ◐ 90 seconds
1 · Adding tool spans took visible wall clock from 32.4% to 38.2% and left the diagnosis at 0 of 5. Why is that the most important rung on the ladder?
Framework integrations hand you tool spans early, so they are what teams ship first. On this incident the missing 4.60 seconds was entirely inside a retrieval step that nobody had wrapped, and no amount of tool detail reaches it. The lesson is not that tool spans are useless - it is that coverage has to be chosen against the questions you need to answer, before you know which step will break.
2 · At rung 8 you capture every prompt and completion. Retention goes from 4.6 GB to 48.1 GB, a factor of 10.5, and the PII flag flips to true. The diagnosis stays at 5 of 5. Why does it not improve?
Trace the causal chain and the last link has no telemetry signal at all, because at that point nothing is failing: given stale inputs, the model produced an accurate summary of stale inputs. Content capture is the lever that feels most like debugging and, on this class of failure, contributes nothing while multiplying your retention and converting a telemetry store into a system holding raw customer text.
3 · The p95 latency dashboard was green and no alert fired, yet the warehouse query had timed out. What made the system structurally incapable of telling anyone?
No threshold can fire on a signal that was never emitted. The two missing lines are set_status(ERROR) and record_exception(e) in the except block, and their absence is what separates a system that is degraded and knows it from a system that is degraded and reports itself healthy. This is also why session b8's alerting work has to sit on top of coverage work rather than beside it.