Where tonight sits in the builder track
Eight sessions, one artifact: a reliability contract for one pipeline. Session 1 measured it, session 2 set the target. Tonight adds the two lines that decide whether anybody sleeps: the error budget the target implies, and the alert policy that pages when the budget is burning faster than the month can afford. Session 4 writes the runbook the page points at. Everything tonight is arithmetic over the partition log, and all of it runs as SQL.
★ What you walk out with tonight. The budget for your target in bad partitions, and the per-hour rate that would spend it exactly by night 30. A burn-rate query over two windows that pages on the rising edge and never twice inside twelve hours. And the comparison that ends the "alert on everything" argument in your team: 35 pages against 5, for the same five incidents, with the count unchanged whether the target is 95 or 99.9.
LiveWhy the pager should watch the budget, not the partition3 min▶
A bad partition is a symptom. An incident is the thing that produces symptoms, usually many of them in a row. Night 9 at Harborlight was one incident - the source API was down - and fourteen symptoms, one per hour. A policy that pages on symptoms woke the on-call fourteen times to tell them the same fact. The first page was information. The other thirteen were the pager confirming it had not been fixed yet, which the on-call, awake and staring at the source's status page, already knew.
Burn rate reframes the question the pager asks. Not "did a partition fail" but "is the budget being spent at a rate that will break the promise". That question is true for the whole of an incident and false in between, so it has an edge: a moment it becomes true. Paging on that edge pages once per incident, which is what "actionable" means when it is defined mechanically.
Harborlight's on-call rota had a rule nobody had written down: after the third page in an hour, mute the channel and check the dashboard in the morning. It was a reasonable response to 8.2 pages a week. It also meant that on night 21, the flapping connector's eight scattered failures were muted after the third, and the partial load on night 27 was found by the settlement team. The policy had trained the humans to stop listening.
Self-studyThe vocabulary tonight adds3 min▶
- Error budget - one hundred percent minus the target, times the events in the window. At 97 percent over 720 partitions: 21.6 bad partitions.
- Sustainable rate - the budget spread evenly over the window's hours. 21.6 over 720 hours is 0.03 bad partitions an hour. Spend at exactly that rate and the budget is gone at the stroke of night 30, which is the definition of keeping the promise with nothing to spare.
- Burn rate - the actual spending rate over a window, divided by the sustainable rate. A burn rate of 1 lasts the month. A burn rate of 14.4 exhausts the budget in about two days. A burn rate of 3 exhausts it in about ten.
- Fast window and slow window - two lookbacks over the same hourly series. The fast one catches a burst; the slow one catches a smoulder that no single hour would flag.
- Rising edge - the hour the paging condition turns from false to true. The page fires here and nowhere else.
- Suppression - a minimum gap between pages. Twelve hours here: a second rising edge inside that gap is the same incident wearing a new hour.
- Actionable - the first page inside a real incident. Defined mechanically so the two policies can be compared without arguing about how the human felt.
Burn rate is budget spent per hour, relative to the rate that lasts the month
8 min liveThree numbers, each derived from the one before. The budget comes from the target. The sustainable rate comes from spreading the budget over the window. The burn rate is what you actually spent in a window, divided by what you could afford to. Nothing here needs more than division.
LiveThe budget, and the rate that spends it exactly by night 303 min▶
The budget is (1 - target) * 720 partitions. At 97 percent that is 21.6; at 95 it is 36; at 99 it is 7.2; at 99.9 it is 0.72. Fractions are fine - the budget is a ceiling, not a count of things that happen.
The sustainable rate is the budget divided by the hours in the window: 21.6 / 720 = 0.03 bad partitions an hour at 97 percent. It is the same as 1 - target, which is a useful sanity check. A pipeline that failed exactly 3 percent of its hours, evenly, would arrive at night 30 with a budget of zero and a promise intact. Nothing fails evenly, which is why the next number exists.
LiveBurn rate: a ratio of rates, and what 1, 3 and 14.4 mean3 min▶
Over any window, the burn rate is (bad partitions in the window / hours in the window) / sustainable_per_hour. It is a rate divided by a rate, so it is unitless, and it reads directly as "how many times faster than the month can afford".
- Burn rate 1 - spending at exactly the sustainable rate. The budget is gone at the end of the month and not a moment before. This is the promise kept with nothing left over.
- Burn rate 3 - three times too fast. 720 hours divided by 3 is 240 hours: the budget would be gone in about ten days if it kept up.
- Burn rate 14.4 - 720 divided by 14.4 is 50 hours. About two days. This is the rate that means "something is wrong right now", not "the month is drifting".
At 97 percent, one bad partition in a two-hour window is 0.5 an hour, which is 0.5 / 0.03 = 16.7 times sustainable. A single bad hour already clears 14.4. That is not a flaw; a pipeline promising 97 percent really is in trouble the hour it starts failing, and the window's job is to say so once.
Self-studyWhy page on the rate and not on the partition3 min▶
- A rate has an edge; a partition does not. "Bad partition" is a fact about one hour, and the next hour produces a fresh one. "Burning at 14.4x" is a condition that is true for a stretch and then false, so it has one moment of becoming true. Paging on that moment is paging on the incident.
- The rate carries the target's meaning. A threshold page says a partition failed. A burn-rate page says the promise made in session 2 is at risk. The second one tells the on-call how urgent it is; the first one cannot.
- The rate sees what one hour cannot. Night 21's connector failed eight times, never two hours in a row. No single hour looked like an outage. Over twelve hours the pipeline was burning at several times the sustainable rate, and only a window that long can see it.
- Repeats train people to mute. Thirteen pages saying "still down" are not thirteen pieces of information; they are the reason the fourth page of the next incident gets ignored.
Two windows, a rising edge, and a suppression
9 min liveThe bench's policy, stated as the bench's own choices: a fast window of 2 hours that pages at a burn rate of 14.4 or more, a slow window of 12 hours that pages at 3 or more, a page on the rising edge only - the hour either condition becomes true - and never twice inside 12 hours. Those numbers are tunable. What they produce over Harborlight's month is not a matter of taste.
LiveThe fast window: night 9 at its first hour3 min▶
At hour 0 of night 9 the source API went down and the partition did not land. The two-hour window now holds one bad partition in two hours: 0.5 an hour, 16.7 times the 97-percent sustainable rate, over the 14.4 line. The condition is true, it was false the hour before, and no page has fired in the last twelve hours. One page, at the first bad hour of the outage.
Hours 1 through 13 are also bad, so the condition stays true through all of them. A condition that stays true has no rising edge, so no further pages. When the source came back at hour 14 the window emptied over the next two hours and the condition turned false, ready for the next incident. Under threshold paging the same fourteen hours were fourteen pages.
LiveThe slow window: what turns night 21's eight failures into one page3 min▶
Night 21's connector failed at hours 1, 4, 7, 11, 14, 18, 21 and 23. Never two in a row. The first failure at hour 1 is a rising edge on the fast window, so the page fires there. The question is what stops seven more pages, and the answer is the slow window, in two parts.
- The suppression covers the first stretch. Hours 4, 7 and 11 are inside twelve hours of the page at hour 1. Even where the fast window sees a fresh edge, the policy will not page twice inside twelve hours.
- The slow window holds the condition true for the rest. From hour 4 onward, any twelve-hour lookback holds at least two bad partitions: 2 over 12 hours is 0.17 an hour, 5.6 times sustainable, over the 3 line. So from hour 4 the condition never turns false, which means hours 14, 18, 21 and 23 are not rising edges at all. One night of flapping, one page.
Under threshold paging, night 21 was eight pages spread over twenty-two hours, each one arriving after the previous one had been investigated and closed as a one-off. Nobody saw the pattern until the postmortem laid the hours out in a row. The twelve-hour window is that postmortem, computed live.
LiveThe month under both policies3 min▶
| Alert policy | Pages in 30 nights | Pages per week | Actionable | Incidents paged |
|---|---|---|---|---|
| No alerting | 0 | 0 | - | 0 of 5 |
| Page on every bad partition | 35 | 8.2 | 5, which is 14 percent | 5 of 5 |
| Multi-window burn rate | 5 | 1.2 | 5, which is 100 percent | 5 of 5 |
Both policies catch all five incidents. The burn-rate pages land on nights 4, 9, 15, 21 and 27, each at the first bad hour of its incident at the 97 target. The thirty pages that separate the rows are the ones that told the on-call something they already knew: thirteen of them on night 9 alone.
Self-studyTwo honest notes: the numbers are choices, and the target is a different lever4 min▶
- 2 hours, 12 hours, 14.4x, 3x and 12 hours of suppression are tunable. They are the bench's own choices, not a law. A shorter fast window pages sooner and risks paging on a single freak hour; a longer slow window catches slower smoulders and takes longer to clear. Change them, rerun the month, and keep whichever set pages once per incident on your own history.
- The page count does not move when the target does. Rerun the month at 95, 99 or 99.9 and the burn-rate policy still produces five pages and the threshold policy still produces 35. The target sets the sustainable rate, which shifts exactly which hour a page lands in, but a burst is a burst at any target and the rising edge is still one edge. The target decides the budget; the policy decides the pages. They are separate levers, and tightening the first because the pager is noisy is the most common mistake in the field.
- Every bad partition weighs the same here. A corrupt settlement total is worse than a late partition, and a mature budget would weight them. Add that once the unweighted version is running.
- The suppression is a statement about incidents, not about noise. Twelve hours says "two rising edges this close are the same incident". If your incidents are shorter than that, shorten it.
Replace threshold paging with a burn-rate policy
★ 20 min · everyone buildsUse the partition log, the good-event definition from session 1 and the target from session 2. The steps below use 97 percent so the Harborlight numbers in the comments match; substitute your own target and the shape of the query does not change. Bring the incident list from session 1's homework for the last step.
Compute the sustainable per-hour rate for the target. Budget over 720 hours. At 97 percent: 21.6 partitions, 0.03 an hour. Write it down; every burn rate tonight is divided by it.
Compute the 2-hour and 12-hour bad counts per partition. Two window sums over the hourly series, one covering the current hour and the one before, the other the current hour and the eleven before.
Compute both burn rates. Bad count over hours in the window, divided by the sustainable rate. The condition is true when the fast rate is at least 14.4 or the slow rate is at least 3, with at least one bad partition in the window.
Flag the rising edge and apply the 12-hour suppression. A rising edge is the condition true now and false the hour before. A page is a rising edge with no page in the previous eleven hours.
Count pages and compare with threshold paging. Threshold pages are the bad partitions themselves. Harborlight: 35 against 5, which is 8.2 a week against 1.2.
Count how many of each are the first page inside an incident. Join pages to the incident table, number them within each incident, and count the ones ranked first. Harborlight: 5 of 35 and 5 of 5.
One approximation, named. The suppression in the pages step checks for a rising edge in the previous eleven hours rather than for a page, because "since the last page" depends on earlier page decisions and needs a recursive query or a loop to state exactly. On Harborlight's month the two give the same five pages. If your history has two rising edges inside twelve hours where the first was itself suppressed, the exact version pages on the second and this one does not; write the recursive version once that case shows up.
Now change only the target. Replace 0.03 with 0.05, 0.01 and 0.001 and rerun. The budget changes, the exhaustion night changes, and the page counts stay at 35 and 5. If somebody on the team wants a stricter target because the pager is too noisy, this is the query to show them.
Before session 4
30 min- Run both policies over your own last thirty nights and fill in the two-row table: pages, pages per week, actionable, incidents paged. If burn rate misses an incident that threshold caught, look at the incident's shape before touching the windows; a single bad hour in an otherwise clean day is a question for the target, not the policy.
- Rerun the burn-rate policy at two other targets. Write down the page count each time. It should not move; if it does, find the incident whose first page changed hour and work out why.
- Try one alternative pair of windows - say 1 hour at 14.4x and 6 hours at 6x - and count pages again. Keep whichever set pages once per incident on your history, and write the reason in the contract next to the numbers.
- For each of the five pages, write the first sentence the on-call should read when it arrives. Session 4 turns those sentences into runbooks.
Three questions
Harborlight moves the target from 97 to 99 percent and reruns the burn-rate policy over the same month. What happens to the page count?
Threshold paging stays at 35 and burn rate stays at 5 across all four targets. What moves is the exhaustion night: 15 at 97, 9 at 99. Tightening the target because the pager is noisy pulls the wrong lever.
Night 21's connector failed at hours 1, 4, 7, 11, 14, 18, 21 and 23, never two in a row. Why does the burn-rate policy page once rather than eight times?
Two bad partitions in twelve hours is 5.6 times the 97-percent sustainable rate, over the 3 line. The slow window is what sees a scattered night as one incident.
A burn rate of 14.4 over the fast window means what, at any target?
Burn rate is a ratio of rates: actual spending per hour over sustainable spending per hour. 720 hours divided by 14.4 is 50 hours. At 97 percent, one bad partition in two hours already reads 16.7.