Why cost and latency belong on the span, not on a bill
Your provider invoice is a single true number that answers no question you actually have. It cannot tell you which tenant, which feature or which failing retry spent the money, and it cannot tell you whether last Tuesday's slow hour was decode time or a tool backend. Everything useful about cost and latency is a per-span fact, and this session is about getting those facts right - because the two most natural ways to compute them are both wrong, in opposite directions.
input_tokens: 50, two latency definitions that need separate SLOs, and two derived quantities - model_time_fraction and orchestration_overhead - that between them decide whether your next optimisation is a model swap, a prompt edit or a missing span.
The same tokens, counted at two depths 6 min live
The agent conventions ask invoke_agent to carry aggregate gen_ai.usage.* for the whole run, and they ask each child chat span to carry its own. Both are correct and both describe the same tokens. Nothing in the data warns you about this, so the first cost query anybody writes - sum the usage attribute across all spans in the trace - returns exactly twice the truth.
Both correct answers have a failure mode, and choosing between them is a real decision rather than a preference. Leaf-only is robust to a framework that forgets to aggregate, and it is the version that keeps working when a new tool or a nested sub-agent appears - but it depends on you having instrumented every model call, including any that a third-party library makes on your behalf. Root-only is a single cheap read per trace and it is the version your finance queries will want, but it silently under-counts the moment a model call happens outside an instrumented agent span, which is exactly what happens the first week somebody adds a background summariser.
# leaf-only: robust to a missing aggregate, needs full model-call coverage leaf_in = sum(s.attr["gen_ai.usage.input_tokens"] for s in spans if s.op == "chat") leaf_out = sum(s.attr["gen_ai.usage.output_tokens"] for s in spans if s.op == "chat") # root-only: one read, but it trusts the agent span to enclose every call root_in = root.attr["gen_ai.usage.input_tokens"] assert leaf_in == root_in, "coverage gap or a double-counted aggregate"
Self-studyTwo more places the additivity assumption quietly breaks3 min read▶
- Parallel tool calls. Durations do not add when spans overlap. Two tool spans of 2s each running concurrently under a 2.1s parent are not 4 seconds of anything - the parent is right and your
SUM(duration)is meaningless. Any latency figure you compute from a trace has to know whether siblings are sequential or concurrent, which means reading start timestamps and not just durations. - Retries inside one span. The conventions put the whole logical operation, including automatic retries, in a single span. That is good for readability and it means a span's duration can contain two or three attempts, and its token counts should be the billed total across them. So a slow span is not necessarily a slow model - it may be a retried model. If you want to tell those apart you have to emit a retry-attempt count yourself, because nothing in the data model will do it for you.
- Operations that must not report tokens. The conventions specifically tell instrumentations not to report token usage on
fetch_responseoperations. Sum blindly over every span with a usage attribute and you will eventually pick up something that was never meant to carry one.
Four counters, because one of them lies 7 min live
Prompt caching changes what the word "input" means, and it changes it silently. On a cached request the field named input_tokens counts only the tokens that arrived after the last cache breakpoint. Nothing errors, nothing warns, and the number is genuinely correct for what it measures. It is simply not the number your cost dashboard thinks it is reading.
gen_ai.usage.cache_read.input_tokens and gen_ai.usage.cache_creation.input_tokens as included in gen_ai.usage.input_tokens. The provider API's own input_tokens field excludes them. Those two sentences describe different arithmetic, so a naive field-to-attribute mapping produces a span that is wrong by the size of your cache. Do the addition in the instrumentation and emit the sum, keeping the two cache counters as their own attributes.
u = resp.usage span.set_attribute("gen_ai.usage.cache_creation.input_tokens", u.cache_creation_input_tokens) span.set_attribute("gen_ai.usage.cache_read.input_tokens", u.cache_read_input_tokens) span.set_attribute("gen_ai.usage.output_tokens", u.output_tokens) # the spec wants cache tokens INCLUDED here, the API field excludes them: add them back span.set_attribute("gen_ai.usage.input_tokens", u.input_tokens + u.cache_creation_input_tokens + u.cache_read_input_tokens)
The caching economics, and why they are a latency problem too
Caching is priced on both sides of the transaction, which is what makes it possible to lose money on it. Writing a cache entry costs more than sending the tokens uncached; reading one costs a tenth. All three multipliers below were read on 2026-08-13.
| Operation | Multiplier vs base input | What it should look like on a span |
|---|---|---|
| 5-minute cache write | 1.25x | cache_creation.input_tokens non-zero. Break-even arrives after a single read, so one follow-up turn pays for it |
| 1-hour cache write | 2x | cache_creation.input_tokens non-zero and expensive. Needs two reads to break even, so it is a bet on the session continuing |
| Cache read (hit) | 0.1x | cache_read.input_tokens non-zero, cache_creation zero. This is the span you want to see most of the time |
Now the part teams do not expect: the time-to-live runs from the start of the writing request, and streaming time counts against it. A response that takes four minutes to stream has spent four of its five cached minutes before the user has even read it, leaving roughly one minute for the follow-up turn to land. That is a latency problem causing a cost problem, discoverable only if your spans carry both durations and cache counters, which is the entire thesis of this session in one sentence.
Self-studyThree more silent cost regressions, all invisible without the quartet4 min read▶
- The minimum cacheable prefix is enforced without an error. Depending on the model it sits somewhere between 512 and 4,096 tokens, and a request below it is processed with no caching and no error returned. Your code says it is caching, your bill says it is not, and the only signal is
cache_creation.input_tokensandcache_read.input_tokensboth sitting at zero on a request you believe is cached. That is a silent multiple-x cost regression whose sole detector is an attribute you chose to log. - The invalidation hierarchy runs tools, then system, then messages. Changing tool definitions invalidates the whole cache. "We added one tool" is a plausible sentence in a standup and a doubling of your input bill on the same traffic. The visible signature is a step change in cache hit rate on a deploy, which is why hit rate deserves step-change detection rather than a threshold (session b8).
- The tokenizer changed under a stable price. Newer models use a newer tokenizer that produces roughly 30% more tokens for identical text. Same posted price, same traffic, ~30% more spend. Your dollars-per-token chart looks perfect; the chart that moves is tokens-per-request, and you only have that chart if tokens are on spans.
- The contrast worth knowing. One major provider prices cache writes explicitly, so a badly placed breakpoint costs money. Another caches automatically with no write premium, so a prompt whose prefix varies - a timestamp at the top, a username in the system message - silently forfeits the entire discount instead. Both failures look identical on a dashboard that only tracks total spend.
Two definitions, two SLOs, two derived quantities 7 min live
"The bot is slow" is not one measurement. There are two published definitions of model latency and they move for different reasons, so a single p95 on a single number will hide whichever of the two is currently broken.
| Baseline latency | Time to first token | |
|---|---|---|
| What it measures | The model's own time to process the prompt and produce the response, without reference to tokens per second | The gap between sending the prompt and the first token of the response arriving |
| Dominated by | Decode. Sensitive to output length | Prefill. Sensitive to input length |
| Moves when | You ask for a longer answer, or switch to a slower-decoding model | Your context grows, a cache miss forces a full prefill, or the provider queues you |
| Where it lives | gen_ai.client.operation.duration as a metric, span duration on the trace | gen_ai.response.time_to_first_chunk on the span when streaming, gen_ai.client.operation.time_to_first_chunk as a metric |
| SLO | Separate. This is the one users feel on completion | Separate. This is the one users feel as responsiveness |
Streaming belongs in this table only as a caveat. It improves perceived latency and changes nothing at all about total time, so a streaming rollout will move your TTFT numbers and leave your duration numbers exactly where they were. Treat it as a product decision rather than a performance fix. The published order of latency levers is deliberate and it comes with an explicit warning against optimising prematurely: model choice first, then token length in both directions, then streaming.
The two quantities worth computing on every trace
Neither of these is in any specification. Both are trivial arithmetic over spans you already have, and between them they answer the two questions that get asked in every performance meeting: would a faster model help, and where is the time we cannot see.
model_time_fraction = sum(chat span durations) / root duration. A low fraction means the problem is tools, network or orchestration, and a faster model will not help. The quietly remarkable thing about this number is that it is computable on day one from auto-instrumentation alone - the chat spans are the ones your SDK gives you free, and the root duration is the one thing you always know. It came out at 32.4% both before and after the coverage work above. Which means the answer to "should we migrate to the fast model" was available on the very first trace, and it was no.
orchestration_overhead = root duration - (sum of sequential children + max of parallel children). This is the coverage thesis of session b1 restated as a metric. Large overhead does not mean your orchestration layer is slow; it means your orchestration layer is undescribed, and the honest reading of a big number here is "there is a span I have not written yet". Note the shape of the formula: sequential children add, concurrent children take a maximum, so you cannot compute it without start timestamps.
model_time_fraction = sum(s.dur for s in kids if s.op == "chat") / root.dur # sequential siblings add, concurrent siblings take a max covered = merged_interval_length([(s.start, s.start + s.dur) for s in kids]) orchestration_overhead = root.dur - covered
High duration is two different bugs 4 min live
This is the smallest, most useful rule in the session, and it is the reason tokens belong on spans rather than only on a billing dashboard. A slow span with a lot of output tokens and a slow span with almost no output tokens are unrelated problems with unrelated fixes, and duration alone cannot tell them apart.
set_attribute calls, and it is why the token attributes are Recommended rather than optional in the conventions.
Self-studyWhat multiplies underneath all of this: agents are expensive by design3 min read▶
One published production account puts agent workloads at roughly 4x the tokens of a chat interaction and multi-agent systems at roughly 15x, with token usage alone explaining about 80% of the variance in performance on one browsing benchmark. State the caveat when you use those numbers: they describe one company's own research feature around 2025, not a law of nature. They are a well-documented order of magnitude from a real production system and nothing more.
Two consequences for your span design. First, if a 15x token multiplier is on the table, per-span attribution stops being a nice-to-have and becomes the only way anyone will approve the architecture. Second, the denominator matters more than the numerator: cost per successful task is mean attempt cost divided by success rate, so a 60% success rate multiplies your effective cost by about 1.67x, and the failed attempts are the expensive ones because failures run long before giving up. Session a4 does that arithmetic properly for a leadership audience; what you owe it from here is the per-span data it needs.
Other line items that show up as unexplained spend and are worth a span attribute each, all read 2026-08-13: web search at $10 per 1,000 searches with results re-counted as input tokens on later turns; a single unbounded web fetch running to roughly 125,000 tokens for a 500 kB PDF; code execution at 1,550 free container-hours per organisation per month then $0.05 per hour, billed when files are attached even if the tool is never called; a 1.1x multiplier on every token category when you pin inference geography; tool-use system prompt overhead of 286 to 804 tokens per request that never appears in your own prompt; and batch processing at 50% off input and output which stacks with caching.
Attribute one run, end to end ★ 12 min · everyone builds
Take one real agent run in your own system and produce a defensible cost figure for it. The goal is not the number, it is the list of things you had to fix to get the number.
Write both cost queries from Part 1 - leaf-only and root-only - and run them on the same trace. If they disagree, stop and find out why before you do anything else. That disagreement is worth more than the cost figure.
Add the quartet to your client wrapper: gen_ai.usage.input_tokens as the reconciled sum, plus gen_ai.usage.cache_creation.input_tokens, gen_ai.usage.cache_read.input_tokens and gen_ai.usage.output_tokens separately. Then compute a cache hit rate. If you could not compute one before, notice that the blocker was never analysis - it was two missing attributes.
Find one cached request in your traffic and check whether input_tokens alone would have described it honestly. On a large cached prefix the gap between that field and your billed input is the size of your document, which for a 200,000-token prefix and a short question is a factor of thousands.
Now price it, twice: once at your current rates, once at rates with a 30% higher token count for the same text. That second number is what a tokenizer change does to you at an unchanged posted price, and it is the cheapest stress test in this course.
Decide whether a faster model would help ★ 8 min · everyone computes
Somebody in your organisation is currently proposing a model migration to fix latency. This exercise settles that argument with two divisions and no meetings.
Pick the slowest trace you can find. Sum the durations of every chat span in it and divide by the root duration. That is model_time_fraction. If it is under about a third, a faster model cannot save more than a third of the wall clock even if it were instantaneous, and you now have a number to say so with.
Merge the child intervals - overlapping siblings union, they do not add - and subtract from the root duration. That is orchestration_overhead. Read a large value as a missing span rather than as slow glue, and go find which step it is.
Split your latency dashboard in two: time to first chunk, and total duration. Give them separate SLOs. Then go back through the last month of latency incidents and label each one as prefill or decode. Most teams find their single p95 was averaging two unrelated failure modes into one flat line.
Finally, take the ten slowest spans and plot duration against output tokens. The two clusters from Part 4 will separate visibly, and each cluster goes to a different team. That plot is a better prioritisation artifact than any latency percentile.
orchestration_overhead is 4.60 seconds of 6.80, and where the cost per run becomes knowable at $0.0180 only once the token attributes exist. The arithmetic you built today is what makes the seventh rung of that ladder possible.
Try it yourself - this week ◐ 45-60 min total
- Audit your cost query for the double count. Read the actual SQL or the actual dashboard definition and establish which depth it sums. If nobody can tell you, assume it is summing everything and that your reported cost per request is roughly twice reality - a satisfying bug to fix, because it makes the number go down.
- Compute a cache hit rate. If you cannot, you have found the two missing attributes, and the fix is four lines in one wrapper. Then look for requests where both cache counters are zero on a prompt you believe is cached - that is the silent minimum-prefix rule, and it returns no error.
- Compute
model_time_fractionacross your ten slowest traces and write the median on a sticky note. Bring it to the next meeting where somebody proposes a faster model. This is the highest ratio of insight to effort in the whole session. - Split TTFT from total duration in whatever dashboard your team actually looks at, and set two SLOs. Then check whether any current alert would fire on a TTFT regression alone. For most teams the answer is no, which means prefill can degrade indefinitely without anyone being told.
- Re-verify every price on this page against the provider's own pricing page and write today's date next to each one. That habit is worth more than the numbers, because the numbers move and the habit does not.
Official sources covered
Every attribute name, multiplier and definition on this page traces to a primary source - the OpenTelemetry GenAI semantic conventions and the provider's own caching and latency documentation - catalogued in this course's source map and read on 2026-08-13. This page covers:
invoke_agent plus per-call usage on each childPart 1 · the trap set up in b3, resolved as sum-at-one-depthcache_read.input_tokens and cache_creation.input_tokens as included in input tokensPart 2 · and the reconciliation the instrumentation has to performfetch_responsePart 1 self-study · why additivity breaksmodel_time_fraction and orchestration_overheadPart 3 · derivations, not specification. No standard defines them - they are arithmetic over spans you already haveThree questions before you go 🎯 ◐ 90 seconds
1 · Your root invoke_agent span reports 3,760 input tokens. Its two chat children report 820 and 2,940. Your cost dashboard sums the usage attribute across every span in the trace. What is it reporting?
The conventions ask the agent span for an aggregate and each model call for its own, so both are present and both are right. Summing across depths counts every token twice, and the error is silent because 7,520 is a perfectly plausible number. Sum at one depth only, say which depth in a comment, and assert the two agree so a coverage gap announces itself as arithmetic rather than as an invoice.
2 · A request sends a 200,000-token cached document plus a 50-token question. The span reports gen_ai.usage.input_tokens: 50. What has gone wrong?
The measurement is honest and the interpretation is not. Billed input here is 200,230 tokens, of which 200,000 were read from cache at a tenth of base rate. Keeping the two cache counters separately is also the only way a cache hit rate can exist at all, which is precisely why the conventions carry both of them rather than a single combined figure.
3 · A trace has a root duration of 6.80s and its chat spans total 2.20s. Someone proposes migrating to a faster model to fix latency. What do you say?
This is the cheapest decision in the session: two durations and one division. The pleasant surprise is that model_time_fraction needs no instrumentation work at all, because chat spans are what auto-instrumentation gives you free and the root duration is always known. The number was available on the very first trace, before any coverage work, and it already said no.
Builder session 4 cheat sheet · pin this
input_tokens, output_tokens, cache_creation.input_tokens, cache_read.input_tokens, and their sum.input_tokens counts only tokens after the last cache breakpoint. 200k cached + a 50-token question reads 50.