Error budgets buy decisions

Most teams hear a reliability target as a threat: a number to stay above. The service-reliability literature reads it the other way round. The distance between the target and 100 percent is failure that has been agreed in advance, and agreed failure is a resource. It can be spent on a risky release, on a migration, on a night when the upstream is down and there is nothing to be done. What it cannot be is ignored, because when it is gone the promise is broken and every further bad partition is a breach.

Reading the target as a permission is what makes it a leadership instrument. A threat produces defensiveness and quiet re-measurement. A budget produces a question the team can answer every Monday: how much have we spent, how much is left, and what does that mean for the change waiting to ship?

• 720 events. 24 hourly partitions a night for 30 nights. A partition is good when it landed within its hour, its totals reconcile and all three shards arrived.
• 35 bad events, giving 685 good and 95.14 percent achieved.
• Five incidents. Night 4, a late load, 3 bad partitions. Night 9, an upstream outage, 14. Night 15, a corrupt batch, 6 partitions on time and wrong. Night 21, a flapping connector, 8 scattered across the night. Night 27, a partial load, 4.
• Cumulative spend after each incident: 3, then 17, then 23, then 31, then 35. The exhaustion night for any budget is the first incident whose cumulative total passes it, which is why every candidate promise below runs out on an incident night or not at all.
• Every bad partition counts the same. A corrupt settlement total is worse than a late hour, and a mature budget would weight them. Unweighted is where every team starts, and it is enough to make the four decisions tonight.

Multiply the window by one minus the target. Harborlight's window is 720 partitions, so 95 percent allows 36 bad partitions, 97 percent allows 21.6, 99 percent allows 7.2 and 99.9 percent allows 0.72. A fraction of a partition is a real budget: it means the promise is broken by the first bad hour of the month.

Then spend the budget in the order the month actually happened. Night 4 costs 3. Night 9 costs 14. Night 15 costs 6. Night 21 costs 8. Night 27 costs 4. The budget is exhausted on the first night where the running total passes what was allowed, and from that night to night 30 every further bad partition is a broken promise rather than a spent one.

Thirty-six bad partitions allowed, thirty-five spent, 3 percent of the budget left at night 30. That is one partition of headroom, which is to say the month was kept by the width of a single late hour. As a business decision, 95 percent says: we promise what we have demonstrably delivered, and improvement is funded as a separate line rather than smuggled in as a promise.

The objection is always that 95 sounds low. It is the only row in the table where the team's word held for thirty nights. A promise that sounds better and breaks on night 15 is not a higher standard; it is a lower one with better marketing.

A budget of 21.6 partitions, which the month spent by night 15 when the corrupt batch took the running total from 17 to 23. Half the month in breach. As a business decision, 97 percent is affordable only if one whole class of incident is removed before the promise is made: the upstream outage on night 9 alone cost 14 of the 21.6, so a retry-and-hold arrangement with the upstream team that turns an outage into a late load is what buys this target, and it has to be in place before the number is said aloud, not after.

Said the other way: 97 percent is a promise about work that has not been done yet. That can be a fine promise, if the work is funded, owned and dated. Without those three it is a hope with a percent sign.

A budget of 7.2 partitions, gone on night 9 when the upstream outage landed 14 in a single night. Two thirds of the month in breach, and the budget was on track to fail even without the outage: night 4's late load had already spent 3 of 7.2 in the first week. As a business decision, 99 percent is what "basically 99" was all along, a number chosen because it sounded like the kind of number reliable things have.

The cost is not the breach itself. It is what happens in month two, when the team stops reporting against a number it can never meet, the reader stops believing any number the team says, and the pipeline is back to being "reliable" in word only. A target the history cannot support does not raise reliability; it retires the measurement.

A budget of 0.72 partitions, less than one, gone on night 4 by the first late load of the month. On a typical good night Harborlight lands every partition, so 99.9 percent feels like a description of the pipeline rather than a stretch. It is a description of the median night, promised for all thirty, and the first bad hour breaks it before the first week is out.

As a business decision this is the one that looks like ambition and behaves like a resignation: twenty-six nights of the month spent in breach, and a team that learns within the first week that the number on the slide has nothing to do with anything they can affect. The literature's warning against picking a target from current performance is aimed exactly here.

Every pipeline has two queues competing for the same engineers: changes the business asked for, and work that makes the pipeline fail less. Without a budget the choice between them is made by whoever shouted most recently. With a budget the choice is made by a rule agreed before anyone was shouting.

• Budget healthy: ship. The budget exists to be spent on risk, and a team that ends every month with 90 percent of its budget unspent is over-hardening a pipeline the reader was already happy with.
• Budget burning: stop and fix. No releases to the pipeline until the budget is back above the line, and the engineers who would have shipped the change work the incident's action items instead.
• The rule is agreed in advance, by the people who ship and the people who read, and it names the threshold. "Burning" is not a mood; it is a line on the chart that everybody can see.

Describing an incident in budget terms is the single most useful thing a leader can do with the arithmetic, because it turns "there was an outage" into "that cost us two thirds of the month". Against the 97 percent budget of 21.6 partitions:

Night 9 is the honest limit of the whole exercise. The upstream source was down; nothing the Harborlight team did or could have done that night would have landed those fourteen partitions. The budget still pays for them, because the reader did not get her hours and the reader does not care whose fault that was. Some spend is not yours to prevent. It is still yours to promise around, which is why the target has to leave room for the upstream's bad month as well as your own.

The SRE book's first piece of advice on choosing a target is "don't pick based on current performance". It sounds like a contradiction of everything above, since the whole session has been about reading the history. It is not. The trap is picking the number the pipeline happens to produce, whether that is the typical good night or last month's exact figure, without asking two other questions: what does the reader actually need, and what will the team be able to keep saying out loud in month three.

The 99.9 percent row is the trap in its purest form: the current good night, promised as a target. But so is copying 95.14 onto the slide as if a measurement were a decision. The measured number is the ceiling on what can be honestly promised. The target is chosen underneath it, from the reader's sentence and the work that is funded. The book's companion advice is the way out: it is "better to start with a loose target that you tighten" than to start strict and quietly stop reporting. Promise 95, keep it for three months, fund the upstream arrangement, and then tighten to 97 with the history to show for it.

The budget is computed over a window, and the choice of window is a leadership decision disguised as a technical one. A calendar month resets the budget to full at midnight on the first, which teaches every team in the building to ship its riskiest changes on the second. A rolling thirty nights never resets: night 9's outage stays in the window until night 39, and the team has to live with the spend it caused for a full month.

• Rolling makes the budget honest and makes the shipping rule bite continuously. It is the default worth insisting on.
• Calendar is easier to report to finance and is what many platforms compute by default. If that is what you have, know that "budget remaining" on the 2nd of the month means almost nothing.
• Whichever you use, say which, on the slide, every time. Two people quoting budget remaining over different windows are two people about to have an argument nobody can win.