The failure mode is not "we missed the alert"
Session a3 built dashboards that answer questions. Session a4 put a cost on every task. Both of those assume somebody is looking. This session is about the hours when nobody is looking, which is most of them, and about the specific way that goes wrong: not a missing alert, but a channel so noisy that a human made a rational decision to stop reading it.
Somebody already measured this, in an ICU 11 min live
When you argue for fewer alerts, you will be told you are arguing for less safety. You are not, and you do not have to make the case from first principles, because clinical engineering made it with numbers and bodies twenty years before your Slack channel existed.
The ECRI Institute publishes an annual Top 10 Health Technology Hazards list. "Alarm hazards" was ranked number one in both 2013 and 2015. Not a top-ten mention. First place, twice. The 2013 edition states the causal chain plainly, and it is worth reading in the original because every clause of it has an engineering twin:
"With too many alarms sounding... staff may be more likely to experience alarm overload or 'alarm fatigue.' These conditions can lead to ineffective responses or prompt unsafe actions. For example, caregivers may turn down the volume of alarms to an inaudible level, or they may improperly adjust alarm limits outside the safe and appropriate range in an attempt to reduce the number of alarms."
By 2015 the list had narrowed its diagnosis. The hazard was no longer described as too many alarms in general but as inadequate alarm configuration policies, and ECRI notes that it had "investigated several alarm-related deaths and other cases of severe patient harm that could have been prevented had more effective alarm configuration policies been in place." That is a governance finding, not a technology finding, and it is the single most useful thing in this session: the fix they landed on was a written, reviewed policy about what is allowed to alarm and at what limits.
LiveWhy 88.8% is the number to quote in your own review4 min▶
Drew and colleagues annotated 12,671 arrhythmia alarms and found 88.8% of them were false positives, with an inter-rater kappa of 0.86 so the labelling itself is defensible. Sit with the shape of that: the monitors were not broken. They were doing exactly what they were configured to do, on real physiological signals, and nearly nine in ten of their conclusions were wrong.
A naive static threshold on an AI quality metric produces the same class of result for reasons Part 2 works through, and the reason to reach for a clinical study rather than an engineering anecdote is that nobody in your review can dismiss it as an excuse for laziness. The finding is not "alerts are annoying". The finding is that alarm noise produced deaths and the remedy was a configuration policy.
| ICU monitor | Your AI service | |
|---|---|---|
| Signal | Real physiology, correctly measured | Real latency, cost and quality, correctly measured |
| Why it fires wrongly | A limit set for a population, applied to one patient | A threshold set for one mode, applied to a multi-modal metric |
| Documented human response | Volume to inaudible; limits widened outside safe range | Channel muted; threshold raised until it stops firing |
| The fix that worked | A written, reviewed alarm configuration policy per unit | A written, reviewed alerting policy per surface |
| The extra failure mode | None of this kind | Silent failure. Green because nothing is instrumented |
Self-studyWhere the analogy stops being useful3 min read▶
Borrowed evidence has to be returned honestly, so state the limits before somebody else does.
- A monitor is attached to one patient; your alert watches a population. Clinical alarms are per-bed and per-person, which is why 187 per bed per day is intolerable. Your alerts are usually aggregate, which makes them quieter and also makes them blind to the intermittent single-user failure the Beacon incident was made of.
- Their false positives were loud. Yours can be silent. An unattached lead on a cardiac monitor triggers a leads-off alarm. An uninstrumented retrieval step triggers nothing at all, forever. This is the asymmetry that makes AI on-call harder than the ICU case, not easier, and it is why instrumentation coverage itself has to be an alertable class.
- Nobody dies when your chatbot is wrong, usually. Say so. The transfer being argued is about the mechanism of human response to noise, not an equivalence of stakes. Overclaiming here costs you the room.
Five reasons a static threshold misfires on an AI metric 10 min live
Google's SRE workbook already names four generic problems with any threshold alert: precision, recall, detection time and reset time. AI systems add five more that are specific to what you are measuring, and each one is a false-positive factory in its own right.
Self-studyReason 2, in full, because it is the one people refuse to believe3 min read▶
A published experiment ran the same prompt at temperature 0 a thousand times against one model. Those 1000 completions contained 80 unique outputs, the most common of which appeared 78 times. All 1000 were identical through token 102 and diverged at token 103. The cause is not GPU randomness: the kernels are run-to-run deterministic but not batch-invariant, and server batch size varies with other users' load. In the researchers' framing, the other concurrent users are not an input to the system but a nondeterministic property of it.
Two consequences for on-call, both structural:
- Any per-request quality gate is a coin flip with extra steps. If the same input can produce 80 outputs, a threshold that judges one output cannot distinguish a regression from the ordinary spread. Quality alerts have to be a rate over a sample, with the sample size chosen before the alert is written.
- You cannot reproduce a production trace by replaying it, so you must capture enough of it at the time. That is the whole argument for tracing over reproduction, and it is why the coverage decision in session a2 is upstream of every alerting decision here.
Worth knowing when someone proposes pinning temperature to zero as a fix: on recent frontier models the sampling parameters have been deprecated and return an error on non-default values, so that lever is not even available.
Three burn-rate rules instead of thirty thresholds 12 min live
The alternative to a wall of thresholds is not fewer thresholds. It is a different question. Instead of asking "is this number bad right now", a burn-rate alert asks "at the current rate, how fast are we spending the error budget we agreed to", which is the only version of the question a business can act on.
LiveWhat to alert on when success is not a status code6 min▶
Burn rate tells you how to alert. This is what to point it at. Two of these rows are the ones almost nobody has, and they are the two this course exists for.
| Signal | Why it earns a page or a ticket | Route |
|---|---|---|
| Task success rate | Assertion-verified on the output, not the HTTP status. This is the SLI everything else supports | Page |
| Error rate, with 429 split out | Throttling and breakage need different humans and different fixes. Mixed together, both get diagnosed wrongly | Page |
| p95 first-token and p95 total, separately | They move for different reasons. One number for both hides which half is broken | Page |
| Refusal rate | A refusal returns as a normal 200 response with a refusal stop reason. Invisible to error-rate alerting, and a refusal spike is a product outage | Page |
| Cost per hour and cost per successful task | Session a4's denominator. A runaway loop shows here before it shows anywhere else | Page on rate, ticket on ratio |
| Truncation rate | Both max-token stops and context-window-exceeded stops count as truncated. A truncated answer looks successful | Ticket |
| Step and tool-call count distribution | A runaway loop is visible as shape, without opening a trace. The spec's own histogram buckets run to 128 for exactly this | Page above p99 |
| Cache hit rate, step-change detection | A sudden drop means a cache-invalidating change shipped. It is a cost alert that fires the day of the deploy, not on the invoice | Ticket |
| Guardrail trip rate | Both directions matter. A spike is abuse or a prompt regression; a collapse to zero usually means the guardrail stopped running | Ticket |
| Eval-score drift on sampled traffic | Small noisy sample scored by a judge whose own drift you have not measured. Real signal, wrong instrument for waking people | Ticket, never page |
| Instrumentation coverage | Traces missing an expected span type. The alert that catches the silent failure, because a step nobody instruments cannot look slow, empty or failed | Ticket |
| Response model versus requested model | Catches silent alias movement. Your request said one thing, the provider served another, and no other signal shows it | Ticket |
Note what the routing column is doing. Half of these are deliberately tickets. A leader's contribution to on-call health is mostly the discipline of deciding, in advance and in writing, which signals are allowed to wake somebody.
Audit your own channel against the ICU numbers ◐ 10 min · in pairs
This exercise has one rule: no proposals until the count is on the table. Teams argue about alert quality for months without ever writing down how many alerts fired last week, and the count usually ends the argument by itself.
Pull the last seven days from your busiest alerting channel and write two numbers on the board: how many fired, and how many were followed by a human action other than acknowledging or muting. The second number divided by the first is your precision. Compare it to 11.2%, which is what the ICU arrhythmia monitors managed.
For every alert that fired more than three times, ask which of Part 2's five reasons explains it. Most will be reason 3, a single threshold across a legitimately multi-modal distribution. Naming the reason turns "this alert is noisy" into a specific fix.
Ask the on-call rota directly: which alerts have you muted, filtered or stopped reading? Ask it as a design question rather than a compliance question, because the honest answer is the most valuable data in the room and you will only get it once.
Now the uncomfortable half. List the last three incidents your users reported before your monitoring did. For each one, name the signal that would have caught it and whether that signal exists at all. Absences do not appear in an alert audit, which is why they need their own pass.
One page, two columns. Left column: alerts to delete or downgrade to a ticket, with the reason number from Part 2 next to each. Right column: signals that do not exist yet, with refusal rate and instrumentation coverage almost certainly on it. Both columns need an owner and a date, or you will run this same exercise again next quarter with the same numbers.
Write the alerting configuration policy ◐ 12 min · whole group
ECRI's 2015 finding was not that hospitals needed better monitors. It was that they needed alarm configuration policies, reviewed for completeness and clinical relevance. Copy that shape exactly. One page per surface, reviewed on a date, owned by a person.
Name the surface and its SLO. One policy per surface, not one for the company. "The assistant's answer endpoint, 99.9% task success measured by output assertion" is a policy scope. "AI" is not.
Set the burn-rate rules, not thresholds. Take the three rows from Part 3 as written: page at 14.4 over 1 hour and 5 minutes, page at 6 over 6 hours and 30 minutes, ticket at 1 over 3 days and 6 hours. Both windows must exceed before anything fires. Resist the urge to invent your own numbers on day one.
Declare the page list, and declare it short. Everything not on that list is a ticket by default. Write down that default explicitly, because the absence of a stated default is how a channel accumulates thirty alerts nobody agreed to.
Segment before you threshold. For every latency or cost signal, state which dimension it is segmented by: cache hit against miss, single-step against multi-step, tenant tier. Reason 3 is the most common cause of noise and segmentation is its whole cure.
Put a review date on every number. Provider-side change moves your baseline without a deploy of yours, so a threshold with no review date is a threshold that will be wrong and unowned. Quarterly is enough.
Give someone the authority to delete an alert. This is the clause the policy usually lacks and the reason noise only ever accumulates. Adding an alert is a five-minute change anyone can make; deleting one feels like accepting risk, so nobody does it without cover. Provide the cover in writing.
ECRI's remedy was an alarm configuration policy "assessed for completeness and clinical relevance". Your version reads: assessed for completeness, meaning every failure mode in our taxonomy maps to a signal, and for operational relevance, meaning every page maps to an action a woken human can actually take at 3am. If a signal fails the second test, it is a ticket. If it fails the first, it is a gap and it gets an owner.
Try it yourself - this week ◐ 45-60 min total
- Compute your channel's precision for one week: alerts that produced a human action, divided by alerts that fired. Write the percentage in the policy document as the baseline you are trying to move. It is the only number in this session that is about your organisation rather than somebody else's.
- Add refusal rate as a tracked signal, segmented by feature. It arrives as a successful response, so no error-rate alert will ever show it to you, and a refusal spike after a prompt change is one of the most common silent product outages in this class of system.
- Write the instrumentation coverage alert: what proportion of traces on your main path contain every span type you expect. Then look at today's value before you set a threshold, because for most teams the first reading is the finding.
- Convert exactly one existing threshold alert to a burn-rate rule and leave both running for two weeks in parallel. Count how often each fires and how often each was right. One head-to-head comparison persuades a sceptical on-call rota better than the whole ICU dataset.
- Take your loudest alert to whoever owns the rota and ask what they would need in order to agree to delete it. The answer is usually a replacement signal rather than reassurance, and that answer is your next ticket.
Official sources covered
Every quotation and number on this page traces to a primary source catalogued in this course's source map, and the clinical evidence is quoted rather than summarised so you can carry it into your own review unaltered. This page covers:
Three questions before you go 🎯 ◐ 90 seconds
1 · The ICU study found 88.8% of annotated arrhythmia alarms were false positives. What is the transferable finding for your team?
The monitors were not faulty. They measured real physiology and reported it against limits, and nearly nine in ten conclusions were still wrong. That is precisely what a static threshold does to a multi-modal, non-deterministic AI metric. The part to carry into your own review is not the percentage but the response it produced: volume turned to inaudible, limits pushed outside the safe range. Engineers mute channels and raise thresholds for exactly the same reason, and they are being rational both times.
2 · Why must a burn-rate alert require both the long and the short window to exceed threshold?
Reset time is one of the four generic threshold problems and it is the one people forget. With only a long window, an incident that ended twenty minutes ago keeps paging because the hour-long average is still elevated, and an alert that fires after the problem is fixed teaches people that alerts are unreliable. The short window is what lets the alert stand down honestly. Both windows together also solve the two failures in SRE's own arithmetic: a trivial 0.1% blip that pages, and repeated 100% spikes that consume 35% of the budget without ever firing.
3 · Your error rate is flat, latency is normal, and cost per hour is unchanged. Which two AI failures can your monitoring still not see?
This is where the hospital analogy runs out. An unattached cardiac lead sets off a leads-off alarm; an uninstrumented retrieval step sets off nothing, ever. Refusals and empty responses arrive with a success status, so no error-rate alert will ever mention them, and a coverage gap is an absence that conventional monitoring is not built to notice. Those two are the silent failure mode, they are the mechanism behind the Beacon incident, and they are both cheap to add once somebody decides they are alertable classes.