Where session a1 left you, and the single move that follows from it
Session a1 ended with a number: the hours of senior time your last unexplained AI answer consumed. Beacon's was four days, spent proving what was not wrong. Anthropic's postmortem showed the same shape at a company with no resourcing excuse. Today has one job, which is turning "we should see more" into a written list of what specifically, owned by a named person, with the cost stated honestly rather than hidden.
You get the model calls free, and the model calls are rarely the bug 9 min live
There is a structural reason your traces look the way they do, and it has nothing to do with your team's judgement. Auto-instrumentation wraps what the vendor ships, and the vendor ships the model client. So the model call is instrumented by default, everywhere, in every stack - and it is the layer that fails least often.
A · What arrives for free
The model calls. Which model, how many tokens, how long it took, what finished it. Genuinely useful, genuinely well covered, and almost never the thing that broke.
B · What needs a decision
Retrieval. Caching. Tool arguments. Retry counts. Swallowed exceptions. Guardrail outcomes. Every one of these is where production actually breaks, and every one requires somebody to choose it.
That is the whole asymmetry, and it explains a pattern you have probably seen without naming: teams with real observability spend on their AI stack, whose first serious incident is still unreadable. They have excellent coverage of the safest layer. Nobody chose that. It was the default, and defaults are decisions that nobody signed.
LiveThe one metric that turns this into a standing question4 min▶
There is a single derived number that makes coverage legible to people who will never open a trace: unaccounted time. Take the total duration of a request, subtract the durations of every span you actually emit, and express the remainder as a percentage.
| Unaccounted time | What it means operationally |
|---|---|
| Above 50% | Most of what your users wait for is undescribed. Any incident in that region will be diagnosed by reading code, on a deadline, by your most expensive people |
| 20 to 50% | You can localise a problem to a region but not to a step. Investigations end in plausible hypotheses rather than mechanisms |
| Under 10% | Duration questions are answerable off the waterfall. Attribute coverage becomes the next constraint, and that is a different decision |
Ask for this number once per quarter, per AI feature. It requires no new vocabulary from the person answering, it cannot be gamed without genuinely improving coverage, and it is one of very few observability metrics that a non-engineer can interpret unaided. Session b1 walks the builder track through computing it.
What each missing instrument makes your team believe 8 min live
A coverage gap does not present as an absence. It presents as a confident wrong answer, arrived at by competent people reasoning correctly from incomplete evidence. This is the table to bring to a prioritisation meeting, because each row is a specific week your organisation can lose.
LiveRow by row, with the decision each one implies5 min▶
| Gap | The wrong turn it causes | What it costs before you notice |
|---|---|---|
| No retrieval span | "The model hallucinated" - so you change model, or tighten the prompt, or add a guardrail. The model was fine. It had nothing to ground on | A model migration. Weeks, plus a confident negative result that means nothing |
| No retry counter | "This model call is expensive" - so the cost conversation becomes a pricing conversation. It was one call, billed four times | A vendor negotiation aimed at the wrong number, while the retry storm continues |
| No cache read versus create split | "Our cost per token is stable" - so nobody looks. A cache write and a cache read are priced very differently, and one number cannot represent both | A cost model wrong by a large multiple, in whichever direction flatters you |
| No guardrail outcome | "Users only ask easy questions" - so the roadmap optimises for the questions you can see. The hard ones were refused, silently, and never reached a log | A product strategy built on a survivorship-biased view of demand |
| No status on a caught exception | "The request succeeded" - because it did, from the caller's point of view. The fallback path ran and nobody was told | Beacon. Four days, and a week of executives acting on last Monday's numbers |
The 12.5x, stated precisely so nobody over-claims it. Anthropic's published prompt caching multipliers are 1.25x base input cost for a five-minute cache write and 0.1x for a cache read, read 2026-08-13. The ratio between those two token classes is therefore 12.5x. If your telemetry reports one undifferentiated input token count, your per-token cost model can be wrong by that factor for the tokens in question. This is arithmetic on published multipliers, not a published statistic, and it is a leading indicator rather than a bill - session a4 does the full cost attribution and b4 does it per span.
The 6%, stated precisely too. It is the refusal rate in the Beacon teaching case, chosen to be unremarkable. Any real number is your own to measure, and the reason to measure it is that a refusal typically returns as a successful response, so it is invisible to error-rate alerting by construction. Session a5 makes refusal rate a first-class alerting signal.
The best-resourced possible team ran green evals and missed a bug affecting 16% of requests at peak. In Anthropic's published postmortem of three issues, September 2025, a context-window routing error affected a peak of 16% of Sonnet 4 requests in the worst hour, and approximately 30% of Claude Code users who made requests in that window had at least one message routed to the wrong server type. The benchmarks, in their words, "simply didn't capture the degradation users were reporting".
Two details make this the most useful case available for a coverage argument:
- The reason detection failed was partly a virtue. Claude recovers from isolated errors, so an aggregate score over many requests barely moved while individual users were visibly harmed. Resilience suppresses exactly the signal an offline benchmark reads.
- The remediation went towards production coverage, not smarter offline testing. More sensitive evaluation on real production traffic, and better tooling for engineers to inspect real requests. When the team with the deepest possible eval investment finds a failure class its benchmarks cannot see, it buys coverage on production traffic instead of buying more benchmarks.
Use this in exactly one way and no further: it is evidence that offline testing has a structural blind spot for intermittent, absorbed, route-dependent failures. It is not a benchmark number, not a guarantee about any product, and not a claim about what your own rates will be. The Bedrock and Vertex figures - 0.18% and under 0.0004% at peak - are the reminder that if you multi-home, one platform's telemetry does not represent the others.
Coverage is a measurable property, not a feeling 8 min live
Everything above is an argument. This is the part that turns it into an operating practice, and it is the most under-used idea in AI observability: write down the list of span types a healthy trace of this feature is supposed to contain, then let your monitoring alert when a trace arrives without one of them.
Self-studyHow to write an expected step list that survives contact with reality4 min read▶
The failure mode of this practice is a list so detailed that it is wrong within a fortnight and gets switched off. Four rules keep it alive.
- List span types, not span counts. An agent loop legitimately runs three tool calls today and eleven tomorrow, so counts flap. "At least one retrieval span" is stable across both. Runaway loops are caught by a different signal, which is a5's subject.
- One list per feature, not one per service. The expected shape of a document assistant is not the expected shape of a routing classifier. A single global list is either so loose it catches nothing, or so tight it fires constantly.
- Version the list with the feature. When a step is deliberately removed, the list changes in the same change set. If updating the list is a separate ticket, it will lag, and a lagging list generates false alerts until somebody mutes it permanently.
- Alert as a rate, not per trace. One trace missing a span type can be legitimate - a cache hit path may genuinely skip the database. A sustained share of traces missing an expected type is the signal. Per-trace alerting on a non-deterministic system is how you build alert fatigue, and a5 is entirely about not doing that.
Written this way, the artifact is short. Seven lines per feature, owned by the team that owns the feature, reviewed when the feature changes. It is the cheapest thing on this page and the only one that catches its own decay.
What coverage costs, said out loud 5 min live
A session that only argues for more instrumentation is a sales pitch, and your engineers will hear it as one. Coverage is not free, the costs are real, and naming them is what makes the ask credible.
1 · Storage and ingest
More spans per trace means more bytes retained and, on volume-priced platforms, a larger bill. Content capture is the expensive part by a wide margin, and it is a separate decision from structural coverage.
2 · Engineering time
The spans you get free cost nothing. Retrieval, cache and tool-argument coverage is deliberate work by the team that owns the code, and it competes with the roadmap.
3 · Cardinality
Rich attributes on spans are cheap. The same attributes used as metric dimensions multiply time series and can dominate cost. The two look identical to a stakeholder and are priced very differently.
4 · Privacy exposure
Structural coverage - durations, counts, statuses - carries little risk. Capturing prompts, tool arguments and retrieved documents creates legal and security obligations. Do not bundle the two asks together.
Self-studyThe trap where more coverage buys nothing3 min read▶
The course's own trace simulator makes this point against itself, and it is worth knowing before you approve a budget. Adding coverage improves diagnosis sharply up to a point: structural spans, then attributes, then the database and cache children, then error propagation, then identifiers and token counts. At that point every diagnostic question about the Beacon incident is answerable.
Then there is one more lever available - capturing the full message content of every request - and pulling it moves the diagnosis score by nothing at all, while multiplying retention volume by roughly a factor of ten and flipping a personal-data flag to true. The message content was never the problem. The retention numbers there are modelled rather than measured, and labelled as such in the simulator.
That is the honest shape of the curve: steep, then flat, then actively harmful. A leader's job on this page is not to maximise coverage. It is to get onto the steep part of the curve and to recognise the flat part before somebody buys it. Sessions b5 and b10 walk the builder track through both ends of that curve directly.
Write the expected step list for one feature, now ★ 8 min · in pairs, on paper
This is the artifact. Eight minutes, one feature, seven lines or fewer. The point is not to get it right; it is to discover how much disagreement exists in the room about what your own system does.
Take the feature you inventoried in a1 and list every step a healthy request passes through, in order, in business language. Not function names - "look up the customer's plan", "search the policy corpus", "call the pricing service", "compose the answer".
Mark each step with whether it can fail independently. If a step can fail on its own and the request can still return something to a user, that step must be able to say so. Those are the ones that matter, and they are usually the ones nobody instrumented.
Now mark which steps you currently believe emit a span. Do not check yet - guess, and record the guess. The gap between the guess and reality is the most informative number this exercise produces, and it is the number worth reporting upward.
Have somebody verify the guess against one real trace before the next session. Two disagreements is normal. Five means the shape of your system in your head is not the shape on disk, and that is worth knowing on a calm Tuesday rather than a bad one.
Decide who owns coverage, and what happens when it drifts ★ 6 min · decide, do not discuss
Coverage decays by default, for mundane reasons, in every organisation. So the question is not whether it will drift but who finds out and what they are empowered to do. Four decisions, made in the room, written down.
The owner. Does coverage belong to the feature team or to a central platform team? Both work. What does not work is the common third answer, where the platform team owns the tooling and the feature team owns the code and neither owns the gap between them.
The gate. Is a missing expected span type a release blocker, a ticket, or neither? If a new AI code path can ship without appearing in a trace, coverage will decay at exactly the rate you ship, and that rate is your best case.
The budget. Which of the four costs in Part 4 is your actual constraint - storage, engineering time, cardinality, or privacy? Naming the real one prevents the argument where a proposal is rejected on privacy grounds when the objection was really headcount.
The review. When is coverage looked at deliberately, by whom, with what number in front of them? Quarterly, per feature, with unaccounted time and the expected-step-list result is enough. Anything less frequent and the first review will be during an incident.
Before session a3 - this week ◐ 40-50 min total
- Finish the expected step list for one feature and get it verified against a real trace. Bring the two numbers: how many span types the design implies, and how many actually appear. That pair is the whole of your coverage position, and it fits in a single line of a status update.
- Get the unaccounted-time percentage for one real AI request - total duration minus the sum of emitted spans. For most teams it lands above fifty percent, and it is the most persuasive figure in this course precisely because nobody has to trust you about it.
- Ask which of your AI features can currently distinguish "retrieval returned nothing" from "the model ignored the context". If the answer is none of them, you now know why the last hallucination discussion in your organisation went nowhere.
- Read the remediation section of the Anthropic postmortem once more and note the direction: production coverage, not better offline benchmarks. If you make one argument to a sceptical peer this week, make it that one, and hand them the source rather than the summary.
- Name your real constraint from Part 4 out loud to whoever controls it. Coverage proposals do not fail on merit; they fail because the objection everyone had was never the objection anyone stated.
Official sources covered
Every claim on this page is either traceable to a primary source in this course's source map, or explicitly labelled as arithmetic or as a constructed teaching figure. Prices carry the date they were read. This session covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · Your platform team reports excellent trace coverage of every model call in the AI stack. What have they actually told you?
Model call coverage is the one thing nobody has to choose, because auto-instrumentation wraps what the vendor ships. It is genuinely useful and it is the layer that fails least often. Retrieval, caching, tool arguments, retry counts and swallowed exceptions each require somebody to decide they matter - which is why a team can hold real observability spend and still be unable to read its first serious incident.
2 · In the Beacon run, the retrieval step is 4.2 seconds of a 6.8 second wall clock. What changed when a retrieval span was finally emitted?
This is the cleanest illustration of why coverage is a decision rather than a technical problem. The 4.2 seconds were always there and were always the majority of what the user waited for. No code path, no dependency and no configuration changed - the only difference was that somebody decided the step deserved to be described. Uninstrumented time does not look slow. It does not look like anything at all.
3 · A trace arrives with four spans, where your expected step list for that feature says seven. Why is this the highest-value alert on the page?
Every other signal in this course reports on a request that has already gone wrong. Coverage is the one that reports on your capacity to explain the next one, and it decays silently for mundane reasons: a dependency upgrade drops a hook, a refactor moves a call outside its span, a new path ships without the wrapper. B is the trap - a missing span type usually means a missing instrument, not a broken step, and conflating the two produces the wrong ticket. Alert on the rate, not on single traces, or you have built the alert fatigue that session a5 is about avoiding.