A dashboard is a question you froze
Every panel on a wall is a question somebody asked once and then stopped asking. That is the whole value of a dashboard and it is also the whole risk: the question ages, the system changes shape underneath it, and the panel keeps returning a confident answer to a question nobody needs any more. Sessions a1 and a2 established that coverage is a decision. This session is about the second decision - which of the things you cover become an aggregate you watch, and which stay a receipt you go and read.
The Beacon incident is a constructed case built for this course. The attribute names, span names and spec rules quoted on this page are the real ones; the trace itself was not captured from a live system, and the timings, percentages and volumes are computed from that constructed case. Everything in the OpenTelemetry GenAI semantic conventions is still Development status with no tagged release, so treat every attribute name here as a moving target and check the repository before you standardise on it.
What a green hour actually meant 14 min live
Start from the test, not from the tooling. A trace is worth the storage it costs when it can answer three questions about one single request. Those three questions are also the exact list of things the Beacon dashboard could not answer, and the reason it could not answer them was not a bug in the dashboard.
| The question | What answers it | What Beacon's telemetry said |
|---|---|---|
| Where did the time go? | Parent span duration minus the sum of the children. Every second of wall clock should sit inside some span | Only 32.4% of the run was inside any span. 4.60s of 6.80s was unaccounted, and unaccounted time is invisible by construction |
| What did each step get and return? | Attributes, not text: row counts, document counts, ids, scores, cache age | Nothing. The warehouse query returned 0 rows and the cache entry was 19.4 hours old, and neither number existed anywhere |
| Did anything fail, including quietly? | Span status set deliberately on the handled path, not only on the crash path | Status OK throughout. A try / except caught the 4.1s timeout, served the stale cache, and returned a confident answer |
Read the third row twice. The system did not lie to the dashboard. The dashboard asked "what proportion of requests errored" and the honest answer was zero, because nothing errored. The failure had been converted into a success by a well-intentioned piece of error handling, and no aggregate over any number of requests can recover information that was destroyed at the moment it was produced.
LiveWhy p95 is the wrong instrument for this class of failure4 min▶
A percentile is a statement about a population, and it is a deliberately robust one. p95 is built to be insensitive to a handful of odd requests, which is exactly what you want when you are protecting an SLO and exactly what you do not want when four answers in twenty-four are quietly wrong.
There is a second, subtler problem specific to AI systems: latency and cost are legitimately multi-modal. A cache hit and a cache miss are different populations. A one-step answer and a twelve-step agent run are different populations. One p95 line drawn across both either fires constantly or hides an entire mode inside the healthy one. Before you threshold anything, segment it - and the attributes you segment on have to be low-cardinality, which is Part 3.
And the failure Beacon actually had was not a latency failure at all. The stale answer arrived at completely normal speed. There is no percentile of any duration metric that could have caught it, because duration was never the symptom.
Self-studyThe 62% you were not shown3 min read▶
In the constructed Beacon trace the retrieval step takes 4.2s of a 6.8s run - 62% of the wall clock - and in the version that shipped, it had no span. The consequence is worth spelling out for a leadership audience because it is a budgeting argument, not a technical one.
- Your engineers were reasoning about the 38% of the request they could see, and reasoning well. The three days they spent on the model call were not incompetence; they were the rational response to the only evidence available.
- The fix was not a code change, it was a visibility change. Once the retrieval span existed, the diagnosis took minutes. Nothing about the system's behaviour changed at the moment the bug became findable.
- That is the shape of the return on instrumentation spend, and it is why the number to ask your team for is not "how many dashboards do we have" but "what percentage of an AI request's wall clock sits inside a span we emit". For most teams the honest answer is under half.
Metrics, traces and logs do different jobs 9 min live
Vendors sell these three as a bundle and then teams treat them as interchangeable, which is how you end up with a metric that cannot answer a question and a trace store you cannot afford. They are not tiers of the same thing. They answer different questions, they fail in different ways, and their costs scale on different axes.
| Metrics | Traces | Logs | |
|---|---|---|---|
| Unit | The population, over a window | One request, as a tree | One moment, as a line |
| Cost scales with | Distinct time series - the number of label combinations, not the request count | Request count and span count, so retention is the lever and sampling is the tool | Volume of text, which is the least predictable of the three |
| Kept for | A year without argument | Days to weeks, sampled | Whatever your bill tolerates |
| Alertable | Yes - this is what alerts are made of | Rarely directly; you alert on a metric derived from traces | Only via a fragile pattern match on text somebody may rename |
| Fails you when | The question needs an id, or the population hides a mode | The trace you need was the one you sampled away | The interesting thing was never printed, and the loop makes iteration 3 look like iteration 1 |
| Beacon reading | Green. Correctly, honestly green | Would have shown it, if the retrieval span had existed | Showed a caught timeout at INFO, in a stream nobody reads |
Notice what the last row says about the log line. The information was technically present. A timeout was written to a log stream at INFO level, inside an hour that contained several million other lines, with nothing tying it to the specific answer an executive was reading. Data you cannot find is indistinguishable from data you do not have, and this is the honest sense in which the team "had the evidence" and still spent three days.
Cardinality forces the split, and threading undoes it 10 min live
There is one piece of arithmetic behind almost every argument you will have about observability cost, and it is not about how many requests you get. Metrics cost scales with the number of distinct time series, which is the Cartesian product of the label values you attach. Add one unbounded label and you have not made the bill bigger; you have removed its ceiling.
This is the most useful thing on the page for a budget conversation, because it converts a vague "observability is expensive" into a specific rule. You can afford an aggregate over anything bounded - provider, model, operation, tenant if you have tens of tenants, feature if you have tens of features. You cannot afford an aggregate keyed on anything that grows with usage, and the spec authors already made that call for you: not one of the GenAI metrics carries an id in its attribute set.
Which brings us to the second half of Part 3, and to the thing that separates a team who has telemetry from a team who can diagnose with it. A conversation is not a trace.
The mechanics matter here because they are frequently got wrong in a way that costs you the diagnosis. trace_id is scoped to one user turn. A ten-turn conversation is ten traces. If your incident review looks at traces one at a time you will keep concluding "unlucky request" and you will never produce the word that actually changes the fix: intermittent. The join key that reassembles the thread is gen_ai.conversation.id, which the conventions mark Conditionally Required on inference spans.
1. There is no gen_ai.user.id. User-level aggregation is simply not part of the GenAI conventions. The general OpenTelemetry namespace offers session.id and session.previous_id, both also Development, and the platforms fill the gap in their own dialects - OpenInference has user.id, Langfuse has langfuse.user.id and langfuse.session.id. So "cost per user" is a choice you are making, not a standard you are following, and it is one of the small lock-in decisions session a6 is about. The route the spec explicitly blesses is to set these ids in a custom span processor at your application boundary: on the spans, out of the metric labels.
2. When you have no conversation id, you leave it unset. The spec is unusually direct about this, verbatim: "When no identifier for the conversation is available, instrumentations SHOULD NOT populate conversation id. For example, a new UUID, a trace identifier, or a hash of request content SHOULD NOT be used as a fallback value." Fabricating one feels harmless and is worse than empty, because a fabricated id makes every single-turn request look like its own conversation and quietly destroys exactly the intermittency signal you built the field for.
Self-studyThe compaction attribute, and the regression that correlates with nothing2 min read▶
There is a related field worth knowing about because it explains a class of complaint that otherwise looks like superstition: gen_ai.conversation.compacted. It is a positive indicator only - set when a context was compacted, and never set to false.
The complaint it explains sounds like this: "it was good for the first twenty minutes and then it got worse, and we changed nothing." If the conversation's context was compacted partway through, quality can move for a reason that appears in no prompt, no deploy and no model version. Without that flag on the spans, the regression correlates with nothing you record, and the investigation ends in a shrug. With it, the correlation is one query.
Green dashboard, three angry executives ◐ 8 min · in pairs
Read the situation, then commit to an answer before you discuss it. The point of the drill is the order of your asks, not the eventual fix.
It is 10:20. Three executives have independently said that this morning's revenue figures from Insight Bot were wrong. Your latency dashboard is green, your error rate is flat at zero, and your uptime is 100%. Your on-call engineer has spent an hour and found nothing. Your head of data says the warehouse is fine. Nobody can reproduce it.
Decide first, alone. What is the very first thing you ask for? Write it down. Then write down what you would ask for if the answer to the first thing is "we do not have that".
Rank these four asks in the order you would make them: (a) the traces for the three specific answers the executives read, (b) a new dashboard panel for cache age, (c) the percentage of an Insight Bot request's wall clock that sits inside a span you emit, (d) a rollback of last night's deploy.
Now the hard part. Two of those four asks are answerable in ten minutes with telemetry most teams already have, one takes a week, and one is a guess dressed as an action. Which is which, and what does that tell you about what to fund before the next incident rather than during it?
Discuss as a group. If the answer to (a) is "we do not keep traces that long" or "we sample at 1% so those three are gone", you have found a retention and sampling decision that somebody made without knowing they were making it. Who should own that decision, and what would you need to know to make it well?
The request that must be split ◐ 7 min · in pairs
This one arrives by email, from someone reasonable, asking for something impossible. Your job is to give them what they actually need without agreeing to what they asked for.
"Can we get a dashboard showing cost and latency per conversation, broken down by user, so we can see which customers are expensive? Also please add the conversation id and the user email as filters so support can look up a specific complaint."
Name the two questions hiding in one request. One of them is about a population and one is about a lookup. They need different signals and different storage, and answering them as one thing is what makes the request impossible.
Do the cardinality arithmetic out loud for the version as asked: bounded labels multiply out to a few thousand series, then multiply that by every conversation and every user. Say the resulting number as "unbounded" rather than as a guess, because the point is that it has no ceiling rather than that it is large.
Write the counter-proposal in three lines. A distribution on the metrics side with bounded labels only; conversation id and a pseudonymous user key on the spans; a support lookup path that goes to the trace store, not to the dashboard. Then say what each half cannot do, so nobody discovers the limit during an incident.
Decide the uncomfortable one. The email asked for user email as a filter. That is a personal identifier in a telemetry store which usually sits in third-party SaaS with broad internal read access. What do you put there instead, who approves it, and what is your answer when someone says a pseudonymous key makes support slower? Session a6 owns the full version of this argument; today just write your first answer down.
Try it yourself - this week ◐ 40-50 min total
- Audit one dashboard for the three questions. Take the panel your team looks at most and ask which of the three it can answer for a single request: where the time went, what each step got and returned, whether anything failed including quietly. Most fleet dashboards answer none of the three, which is fine as long as everyone knows it.
- Get the coverage number. Ask for one figure: the percentage of an AI request's wall clock that sits inside a span you emit. Not an estimate from the team who wrote the instrumentation - a measurement from a real trace. This is the single number this whole track is about, and for most teams it is under fifty.
- Count the labels on your most expensive metric. Multiply the distinct values of each label together. If any label is an id, a session key, a URL path with ids in it, or anything that grows with usage, you have found a bill with no ceiling and probably an unexplained line item you have been arguing about for months.
- Check whether you can thread. Pick a real multi-turn conversation and try to retrieve every trace belonging to it with one query. If you cannot, you currently cannot distinguish an intermittent bug from an unlucky request - and that distinction changes the fix, not just the story.
- Find your fabricated ids. Ask whether anything in your stack fills in a conversation or session id when one is missing. A UUID per request, a copy of the trace id, or a hash of the prompt all look tidy and all destroy the intermittency signal. The spec's position is to leave the field unset, and it is right.
Official sources covered
The conventions, requirement levels and verbatim spec language on this page trace to the OpenTelemetry GenAI semantic conventions, catalogued in this course's source map. The incident is constructed; the spec rules are not. This page covers:
gen_ai.conversation.id as Conditionally Required, and the verbatim prohibition on fabricating onePart 3 · no UUID, no trace id, no hash of request content as fallbackgen_ai.user.id; session.id and session.previous_id live in the general namespace, and platforms fill the gap in their own dialectsPart 3 · set ids in a span processor at the app boundarygen_ai.conversation.compacted as a positive-only indicatorSelf-study · the quality regression that correlates with nothingTwo honesty flags to carry out of the room. The GenAI conventions now live in a dedicated repository and every span, metric, event and attribute in them is Development with no git tags and no releases, so there is no version to pin and no name to treat as permanent. And the Beacon numbers - 32.4% coverage, 4.60s unaccounted, 19.4-hour cache, 62% of wall clock in retrieval - are computed from a constructed case built to teach the mechanism, not measured from a production system.
Three questions before you go 🎯 ◐ 90 seconds
1 · Your aggregate p95 dashboard stayed green through an hour in which four answers in twenty-four were built on a nineteen-hour-old cache. What is the most complete explanation?
All three matter, and the compounding is the lesson. Fix the coverage gap and an intermittent bug still hides inside the aggregate. Fix the status handling and a step with no span still has no row to turn red. And note that latency was never the symptom - the stale answer arrived at completely normal speed, so no percentile of any duration metric could have caught it. The dashboard was not broken. It was answering a question about the population while everyone in the room had a question about one answer.
2 · Someone asks for a metric of cost per conversation with gen_ai.conversation.id as a label. What do you tell them?
Bounded labels multiply out to something you can budget for - a provider with sixteen well-known values, a model count in the tens, a token type with two. Multiply that by every conversation you will ever have and there is no ceiling at all. This is why the conventions exclude ids from every single metric attribute set: it is not an oversight, it is the guardrail. The business question is real and the answer is a split, because an unbounded attribute is free on a span and ruinous on a metric.
3 · A request arrives with no conversation identifier available. What should the instrumentation put in gen_ai.conversation.id?
Fabrication is worse than absence here, and the reason is diagnostic rather than aesthetic. A conversation spans many traces - one per user turn - and the whole value of the field is that it reassembles the thread so you can see that two turns in four went wrong. Fill it with a per-request UUID or a copy of the trace id and every request becomes its own conversation, the join returns nothing, and you permanently lose the ability to tell an intermittent bug from an unlucky request. The spec's language is explicit, and there is also no gen_ai.user.id to fall back on.
Leader session 3 cheat sheet · pin this
trace_id is one turn; gen_ai.conversation.id is the join key that reassembles the thread, and it is how you learn a bug is intermittent. There is no gen_ai.user.id, and fabricating a conversation id from a UUID, a trace id or a content hash is forbidden by the spec.