The alert is the only part of observability that costs someone their sleep
Everything before this session was about being able to answer a question. This session is about deciding which questions are urgent enough to ask a human at 03:00. That is a different discipline, and it is the one where a badly built observability stack does active harm rather than merely failing to help: a channel that fires 200 times a week gets muted, and a muted channel is worse than no channel because it looks like coverage.
Five reasons a static threshold misfires here 6 min live
Classic APM alerting assumes three things that an LLM endpoint quietly violates: that a status code tells you whether the request succeeded, that a metric is meaningful for one request, and that a single endpoint produces one distribution. All three are false, and each falsehood breaks a different threshold.
LiveSuccess is not a status code, and this is the one that matters most4 min▶
Everything else on this page is a tuning problem. This one is a category error, and it is the reason the Beacon incident held a green dashboard for a full working day. Four documented outcomes arrive as HTTP 200 with no error field set:
- The empty response. A two or three token response with
stop_reason: "end_turn". Documented as happening particularly after tool results. Nothing errors, nothing is logged, and the response has no content in it. - The refusal.
stop_reason: "refusal"withstop_details, returned as HTTP 200 and not as an error. If refusal rate is not its own SLI, a policy change that starts refusing a whole customer segment is invisible to you. - The truncation.
stop_reason: "max_tokens"and"model_context_window_exceeded". Treat both as truncated. The answer is well formed right up to the point where it stops mid-sentence. - The confident fabrication. No signal at all in the response object. The only telemetry that catches it is an assertion against a source of truth, and almost nobody instruments that.
So the first SLI you define is not availability and it is not latency. It is task success, verified by an assertion on the output, and it is the one you attach the fast-burn page to. Everything else on the inventory is supporting evidence.
Self-studyWhy segmenting before thresholding is not optional3 min read▶
One endpoint, four legitimate populations: cache hit and cache miss, single-step and twelve-step. A cache read is priced at 0.1x base input and returns almost immediately; a 1-hour cache write is 2x. A twelve-step agent run can take two orders of magnitude longer than a one-step answer, which is why the spec's own gen_ai.invoke_workflow.duration histogram has a bucket at 7200 seconds.
Mixing those into one p95 gives you a number that is not a property of any real request. Two consequences, both of which you have probably lived through:
- Set the threshold high enough to stop paging on legitimate long runs, and a total collapse of the fast cached path stays under it forever.
- Set it where the cached path lives, and every long run pages you, so within a fortnight somebody widens the limit until it stops firing. That is the documented human response to a noisy alarm and it is how a monitored system becomes an unmonitored one.
The fix is boring and it works: cut every latency and cost SLO by gen_ai.request.model, by cache-hit state, and by a coarse step-count bucket, then threshold inside each segment. The spec hands you the segmentation keys for free because they are the low-cardinality sampling-time attributes from b7.
Google SRE's four failure modes, with the arithmetic 6 min live
Before LLMs there was already a written proof that "alert when the error rate exceeds X for Y minutes" cannot be tuned into correctness. It fails on four axes at once, and every attempt to fix one of them makes another worse. This is the argument that produced burn-rate alerting, and the numbers below are the primary source's own.
Multi-window burn rate, and why both windows matter 7 min live
Burn rate reframes the question. Instead of "is the error rate high", you ask "how fast am I spending this month's error budget, relative to the rate that would exactly exhaust it". Burn rate 1 means you finish the budget exactly at the end of the month. Burn rate 14.4 means you finish it in about two days. That single change of denominator fixes precision, recall and detection time, and the second window fixes reset time.
The canonical table below is the primary source's own, for a 99.9% SLO. Three rules, two of them pages and one a ticket, each pairing a long window with a short window one twelfth its length. The "budget consumed" column is the honest cost of detection: by the time the fast rule fires you have already spent 2% of the month.
| Severity | Long window | Short window | Burn rate | Budget consumed when it fires |
|---|---|---|---|---|
| Page | 1 hour | 5 min | 14.4 | 2% |
| Page | 6 hours | 30 min | 6 | 5% |
| Ticket | 3 days | 6 hours | 1 | 10% |
Write the rule that would have paged at 09:14 ★ 10 min · everyone builds
Beacon's Insight Bot answered every request at 09:14 with a green 200 and a confident summary of nineteen-hour-old data. No availability alert could fire, because availability was perfect. Build the one rule that could have fired, on the one SLI that was actually broken.
Define the SLI as an assertion, not a status. For Insight Bot the assertion is cheap and specific: the answer cites at least one retrieved document, and the freshest cited document is under two hours old. Emit that as a boolean per task. Note what you just did: you turned "the answer was right" into something a counter can hold.
Pick the SLO and write down what the budget buys. 99.9% task success over 30 days allows 0.1% of tasks to fail. Beacon serves roughly 40,000 executive queries a month, so the budget is about 40 failed tasks. Say the number out loud in the SLO review: an SLO nobody can convert into a count of real failures never survives its first argument with a stakeholder.
Precompute the ratio at two window lengths. Do not compute burn rate inside the alert expression from raw counters; you want ratio_rate5m and ratio_rate1h as recording rules so that the alert reads like the table and so both windows come from the same definition.
Write the fast page as an AND of both windows against 14.4x the budget rate, then clone it for 6x over 6h/30m and 1x over 3d/6h as a ticket. Three rules, one SLI. Resist the urge to add a fourth.
Now test the recall case deliberately. Replay a load pattern of 100% failures for five minutes in every ten and confirm the burn-rate rule fires while a "5% errors for 10 minutes" threshold rule stays silent. That test is the whole argument for this session and it takes about ten minutes to build.
Build the alert inventory for Insight Bot ★ 12 min · everyone builds
One rule is not an alerting strategy. The inventory below is the full set this course has taught you to emit, with a decision attached to each row: page, ticket, or neither. Copy it into a document, delete the rows you cannot currently compute, and the remainder is your instrumentation backlog with an urgency already attached.
Fill the third column honestly before you argue about thresholds. A row you cannot compute today is not an alert, it is a ticket in b2 to b7 territory. Most teams find they can compute five of fourteen.
Mark every page-severity row with a burn-rate pair and every ticket row with a plain window. The separation is the point: a page is a claim that a human must act within minutes, and there should be very few of them.
Segment the latency and cost rows before you set any number, using gen_ai.request.model, cache-hit state and a coarse step bucket. If a row cannot be segmented, downgrade it to a ticket rather than shipping a p95 that means nothing.
Write the policy document. One page per surface: what fires, who it wakes, what they are expected to do, and when it was last reviewed. The alarm-fatigue literature is unambiguous that the fix is a written, reviewed configuration policy assessed for completeness and relevance, not a cleverer threshold.
| What you alert on | Signal you compute it from | Page, ticket or neither |
|---|---|---|
| Task success rate | Your own assertion on the output. Never HTTP status | Page · burn rate, 3 rules |
| Error rate, 429 split out | error.type plus status class. Split because a 429 has a completely different fix from a 500 | Page · two separate rules |
| Rate-limit headroom | The anthropic-ratelimit-* header family. Alert on -remaining trending to zero, not on the 429 itself | Ticket, then page at zero |
| TTFT | gen_ai.client.operation.time_to_first_chunk, gen_ai.server.time_to_first_token | Page · its own SLO |
| Total latency | gen_ai.client.operation.duration. A separate SLO from TTFT because prefill and decode move for different reasons | Page · its own SLO |
| Cost per hour | The token quartet plus model price. The only rule that catches a runaway spend inside the hour | Page · hard absolute cap |
| Cost per successful task | Mean attempt cost divided by success rate. Moves the right way when you fix reliability | Ticket · weekly review |
| Refusal rate | stop_reason: "refusal". Arrives as HTTP 200, so it is invisible to every error-rate rule you own | Page · burn rate, segmented |
| Guardrail trip rate | Guardrail span or event count over request count | Ticket · page only on a step change |
| Truncation rate | stop_reason of max_tokens or model_context_window_exceeded. Treat both as truncated | Ticket · leading indicator |
| Eval-score drift on sampled traffic | gen_ai.evaluation.result with gen_ai.evaluation.score.value | Ticket, never a page. The sample is small and noisy |
| Loop and step-count distribution | gen_ai.invoke_agent.inference_calls and .tool_calls. Buckets reach 128 precisely so a runaway is visible without opening a trace | Page on the tail crossing a hard cap |
| Cache hit rate | Share of gen_ai.usage.cache_read.input_tokens. Watch for a step change: a drop means a cache-invalidating change shipped | Page on step change · it is a cost incident |
| Instrumentation coverage | Traces missing an expected span type, and unaccounted time as a share of root duration | Ticket · its own incident class |
gen_ai.response.model vs requested | Both fields on the same span. A mismatch rate above zero means a floating alias moved under you | Page on mismatch rate · see b9 |
Try it yourself - this week ◐ 50-60 min total
- Take your noisiest existing AI alert and compute, from last month's data, how much of the error budget the events it fired on actually consumed. If the answer is under 1%, you have found a precision failure and you can delete the rule this week with evidence rather than with a shrug.
- Run the recall test from Demo 1 against your current alerting: 100% failures for five minutes in every ten, for an hour. If nothing fires, you have just reproduced the primary source's 35%-of-budget example in your own system, which is a far more persuasive artifact than this page.
- Define one task-success SLI as an assertion on output for one AI feature, and write down the monthly failure count the SLO permits. Take that count, not the percentage, to whoever owns the feature.
- Fill in the fourteen-row inventory for one service, honestly, in the third column. Count the rows you cannot compute today. That count is a better measure of your observability maturity than any dashboard screenshot.
- Add the
gen_ai.response.modelversus requested-model comparison as a rule, even if you believe you pin every model. It costs one expression and it is the cheapest insurance in this course. Session b9 explains what it catches.
Official sources covered
Primary sources are marked [P] and secondary [S]. The burn-rate table and the four failure modes are quoted from the Google SRE workbook; the metric names and buckets come from the OpenTelemetry GenAI semantic conventions, which are all Development status with no tagged release, so re-verify the spellings before you build on them.
end_turn, refusal, max_tokens and model_context_window_exceededPart 1 reason 1 · why success is not a status codegen_ai.invoke_agent.inference_calls and .tool_calls to 128, invoke_workflow.duration to 7200sDemo 2 · loop detection without opening a tracegen_ai.evaluation.result and gen_ai.evaluation.score.value as the eval signal in the telemetry specDemo 2 · and it is a ticket, not a pageThree questions before you go 🎯 ◐ 90 seconds
1 · Your monitoring never fires, yet a review shows a series of 100% error spikes lasting 5 minutes every 10 minutes ate 35% of the month's error budget. What is broken?
This is Google SRE's recall failure, in the source's own example. A single-window threshold is asking "is it bad right now, for long enough", and a pattern that is bad half the time never satisfies "for long enough". Burn rate asks a different question: how fast is budget leaving. At 100% errors for half of every ten minutes, budget leaves fast enough that both a 1-hour and a 5-minute window cross 14.4x, and the page fires.
2 · Your automated judge scores 5% of production traces and its mean score drops from 0.81 to 0.74 overnight. What severity does this get?
Eval-score drift on sampled traffic is real signal and it is the wrong shape for a page. The sample is small, quality is noisy at the unit level, and an uncalibrated judge can move for reasons that have nothing to do with your system, which means you would sometimes be paging on the judge's drift. Ticket it, investigate with traces, and keep the pager for assertion-verified task success, refusal rate and hard spend caps.
3 · Which pair of signals belongs on an absolute static threshold rather than a burn-rate rule, and why?
Burn rate exists to spend a budget deliberately: some failure is acceptable, and the question is only how fast. A safety rail has no budget. There is no acceptable rate of thirty-step runaway loops or of a five-figure hour, so the correct response is a circuit breaker at a fixed number, with the notification as a side effect. Keep three of these at most: max steps, max tokens per task, max spend per hour. Everything else gets a windowed rate.
Builder session 8 cheat sheet · pin this
gen_ai.response.model against the requested model.