What the bench measures
A reliability contract has two independent parts: the target, which decides how much failure you have promised to absorb, and the alert policy, which decides when a human is woken. Teams routinely tighten the first hoping to fix the second, and the bench shows why that never works: the page count does not move when the target does.
★ What you walk out with tonight. The month replayed under four targets and three policies, all computed. The number that ends the "just alert on everything" argument: 35 pages, five of them useful, against five pages, all useful, for the same five incidents. And the other trap, computed too: promise the current p50 night for every night and the budget is gone on night 4 of 30.
LiveWhy the target and the policy are separate levers3 min▶
The target is a promise about the month: this proportion of partitions will be good. It sets the error budget, which is the number of bad partitions you can afford before the promise is broken. The alert policy is a rule about the hour: under what pattern of bad partitions does the pager go. One is arithmetic over the window; the other is a decision about the next sixty minutes.
Because they are separate, tightening the target does not reduce pages and loosening it does not add them. What tightening the target does is exhaust the budget sooner, which is a conversation with the business about what to stop shipping, not a conversation with the on-call about sleep.
Harborlight moved the target from 97 to 99 percent after a bad quarter, on the theory that a stricter promise would make the pipeline more reliable. Pages that month: identical. Budget exhausted: night 9 instead of night 15. The pipeline had not changed and the on-call had not slept any better; the team had simply promised something the history said it could not deliver.
Self-studyWhat is real here3 min▶
- The history. 720 hourly partitions, fixed and identical for every learner. 35 are bad, in five incidents: night 4 late load (3), night 9 upstream outage (14), night 15 corrupt batch (6, on time and wrong), night 21 flapping connector (8 scattered), night 27 partial load (4). Achieved compliance 95.14 percent.
- The budget. One hundred percent minus the target, times 720. At 97 percent that is 21.6 bad partitions; at 99.9 it is 0.72, less than one.
- The pages. "Every bad partition" pages on each bad hour. "Multi-window burn rate" pages when the two-hour window burns at 14.4 times the sustainable rate or the twelve-hour window at 3 times, on the rising edge only, never twice inside twelve hours. Those windows and rates are this bench's choices, stated as such.
- Actionable is defined mechanically: the first page inside a real incident. Nothing about the human is modelled.
The matrix, with the numbers it actually produces
10 min liveRead the two tables separately, because they are about two different things. The first is what the target does to the budget. The second is what the policy does to the pager. Nothing in either table depends on the other.
| Target | Budget (bad partitions allowed) | Budget left at night 30 | Exhausted on night |
|---|---|---|---|
| 95 percent | 36 | 3 percent | never |
| 97 percent | 21.6 | 0 | 15 |
| 99 percent | 7.2 | 0 | 9 |
| 99.9 percent, the current p50 night promised every night | 0.72 | 0 | 4 |
| Alert policy | Pages in 30 nights | Pages per week | Actionable | Incidents paged |
|---|---|---|---|---|
| No alerting | 0 | 0 | - | 0 of 5 |
| Page on every bad partition | 35 | 8.2 | 5, which is 14 percent | 5 of 5 |
| Multi-window burn rate | 5 | 1.2 | 5, which is 100 percent | 5 of 5 |
Both policies catch all five incidents. The difference is thirty pages that told the on-call something they already knew: the outage on night 9 alone is fourteen of them. Coverage is the same. Sleep is not.
Live the month
★ 25 min · everyone buildsThe bench is live below. Bars are bad partitions per night, the amber badge is a page, and nights after the budget ran out turn red underneath. Work the steps in order.
Start at 97 percent with no alerting. Budget of 21.6 bad partitions, exhausted on night 15. All five incidents happened and nobody was told; every one was found by whoever opened the dashboard first. That is the baseline every policy is measured against.
Switch the policy to "page on every bad partition". 35 pages, 8.2 a week. Hover night 9: fourteen pages for one outage, the first one useful and the other thirteen telling the on-call the source was still down. Five of 35 pages were the first word of an incident.
Switch to multi-window burn rate. Five pages: nights 4, 9, 15, 21 and 27, each at the first bad hour of its incident. 1.2 a week, all five actionable. Notice night 21, the flapping connector: eight scattered bad hours, one page, because the twelve-hour window sees the pattern the two-hour window cannot.
Now change only the target: 95, then 99, then 99.9. Watch the pages. They do not move. Watch the budget: 3 percent left, then exhausted night 9, then exhausted night 4. This is the whole lesson in three clicks.
Sit on 99.9 percent for a moment. That is Harborlight's typical good night, promised for every night. The budget is less than one bad partition for the month, and the first incident spends it before dawn on night 4. Everything after that is a broken promise, regardless of how well anyone responds.
Go back to 95 percent. Three percent of budget left at night 30, which is one bad partition of headroom. That is what the history actually supports, and session 2 is where you decide whether to promise it as it stands or fund the work that earns 97.
Write down the pair you would sign for Monday. A target the history supports, and a policy that pages once per incident. If you find yourself wanting a stricter target because the pager is too noisy, you have the levers confused.
What the bench leaves out, on purpose. Every bad partition counts the same. A corrupt settlement total is worse than a late one, and a real budget would weight them. Add that first when you build your own; it changes which incidents burn the most and does not change which policy pages sensibly.
Closing the track
45 min- Replay your own last thirty nights from the partition log: budget by night under your current target, and pages under your current policy. Count how many pages were the first word of an incident.
- Compute the target your history actually supports at the 3-percent-headroom level and compare with the one in your contract.
- If you page on thresholds, write the burn-rate version and run both over the same month. Keep whichever pages once per incident.
- Put the two numbers, budget remaining and pages per week, in the monthly report from leader session 6, and keep them there.
Three questions
Harborlight tightens the target from 97 to 99 percent. What happens to the number of pages?
The two levers are independent. Tightening the target changes when the promise breaks; it does not change when the pager goes.
Both alert policies catch all five incidents. Why prefer burn-rate alerting?
Coverage is identical. The difference is entirely in repeats: night 9's outage alone produces fourteen threshold pages, thirteen of them saying the source is still down.
Why is 99.9 percent the trap for a pipeline that achieved 95.14?
The service-reliability guidance is explicit: do not pick a target from current performance. A p50 night promised for every night is a budget of 0.72 partitions and a broken promise by dawn on night 4.