Reporting reliability upward

A reliability report that needs a meeting to be understood is read once, in the meeting, by the people who were there. A report that can be read in ninety seconds by somebody who was not in the meeting is read every month, by everyone it is sent to, and compared with last month's. The second kind is the only kind that changes behaviour over a year, and the test for it is simple: could the person who signs the settlement report read this page and know, without asking, whether the promise was kept?

Three things make a page pass that test. Few numbers, so the reader does not have to choose which one matters. The target next to each number, so the reader does not have to remember it. And the same three numbers every month, in the same places, so that the only thing that changes is the values. Everything a leader is tempted to add to the page, a trend line, a narrative, a score, a colour, makes it need a meeting again.

• Error budget remaining is session 2's arithmetic: 100 percent minus the target, as a count of bad partitions allowed in the window, minus the bad partitions spent. When it goes negative the report says the night it ran out instead of a negative number, because "ran out on night 15" is something a reader can picture and "minus 13.4" is not.
• Pages per week with the actionable share is session 3's on-call design, measured. Pages in the window divided by weeks, and the fraction of those pages that were the first page inside a real incident rather than noise.
• Toil hours is the builder track's session 7: hours spent on work that is manual, repetitive, and would not have been needed if the pipeline were better built. Backfills run by hand, restarts, checks done by eye. Session 5's backfill list is most of it.
• The target beside each is session 2's choice, made from history: 95 percent this quarter, with a plan to earn 97. It is on the page every month so that nobody has to remember it and nobody can quietly forget it.

Each number on the page has a decision attached to it. If a reader cannot say what they would do differently when the number moves, the number does not belong on the page.

Nothing else drives a decision the reader of the report can make. Incident count drives nothing, because most incidents are upstream. Mean time to recover drives the on-call's runbook, which is the team's business and not the report's. Data volume drives capacity, which is a different report. The discipline is not to leave those out because they are unimportant. It is to leave them out because the page has one job.

Somebody senior will ask for one number. It is a reasonable request from someone with forty reports to read, and the answer is still no, for two reasons that survive being said out loud in the meeting where the request is made.

• A single score invites the team to define its own denominator. A score of 87 out of 100 has a formula behind it, and the team that is measured by the score writes the formula. Within two quarters the formula has been tuned so that the score goes up, and nobody outside the team can say what went up. Three numbers with visible denominators, partitions, pages and hours, cannot be tuned without somebody noticing.
• A percentage with no budget behind it is a mood. "Reliability is at 95.14 percent" tells the reader nothing unless they know whether the promise was 95 or 97. Attached to a budget it becomes a fact: 3 percent of the budget left, promise kept, or budget gone on night 15, promise broken for half the month. The score strips off exactly the part that made it a fact.

The sentence for the meeting is short: the page has three numbers because they drive three decisions, and a single score would hide which of the three moved. Then hand over the page, which takes ninety seconds to read, which is less time than the score would have taken to argue about.

The first report is the month of the five incidents, written after the fact, because the first honest report is always a look back at a month the team already lived through. It is also the month that sets the target for the next one.

The pages row deserves a sentence, because it is the one that surprises readers. The five burn-rate pages landed on nights 4, 9, 15, 21 and 27, one per incident, at the first bad hour of each. The old policy's 35 pages were the same five incidents paged once per bad partition. Same month, same incidents, seven times the pages, and only one in seven of them told the on-call anything new. The target row did not change the page count at all; the alert policy did. They are separate levers, and the report shows them separately for that reason.

95.14 percent is a success against a 95 percent target and a failure against 97. The same month, the same 35 bad partitions, the same pipeline, and the word that describes it flips on a number the reader was not shown. A report that gives the achieved figure without the target is asking the reader to guess which word applies, and readers guess in the direction of their last conversation with the team.

• Every number carries its target on the same line. Not in a footnote, not on the previous page, not "as agreed in Q1". Next to it.
• The budget is stated as what happened, not as a fraction. "Ran out on night 15" or "3 percent left at night 30". A reader can picture a night. Nobody pictures 0.86 of a budget.
• When the target moves, the page says so. A quarter in which the target went from 95 to 97 shows both, with the date, so that a jump in "budget remaining" is not mistaken for the pipeline getting better or worse.

Toil is work that is manual, repetitive, devoid of lasting value, and scales with the pipeline rather than with the team. The builder track's session 7 owns the definition, the measurement and the toil budget. What a leader needs from it is one number a month, and three rules about how that number is collected.

• The on-call counts it, weekly, in hours. Not the manager, not at month end from memory. A tally kept during the week is honest; a reconstruction on the last day is a guess with a decimal point.
• It is measured, never estimated. An estimated toil number is the single score wearing a different hat. If the count was not kept one week, the page says "not counted", not a number.
• It is the debt list's first sort key. When toil goes up, whatever the hours went on is the item at the top of session 5's list. Eight hand-run backfills on night 21 are eight hours that a retry with a back-off would have been zero.

Of Harborlight's five incidents, two were upstream: a late load on night 4 and an outage on night 9 that alone cost 14 partitions. A third, night 15's corrupt batch, was a type drifting in a source the team did not own. Three of five, 23 of 35 bad partitions, from outside the pipeline. A leader who promises fewer incidents has promised that the upstream will behave, and the upstream has not been told.

What the team controls entirely is three things, and they are what goes in the promise.

• A design. The on-call is designed, not rostered: burn-rate alerting, a rising edge, 12 hours of suppression, a runbook the tired person can follow. Five pages a month instead of 35 is a design outcome and the team can promise it, because the design is theirs.
• A measurement. Good partitions over total partitions, over a rolling thirty nights, from the reader's side. The team can promise that the number on the page is the number the pipeline produced, every month, with no month skipped and no denominator changed.
• A review cadence. A report every month, sent. A review every quarter, at which the target may move and the debt list is reprioritised. The team can promise the dates.

None of those three is affected by whether the upstream has a bad night. All three are visible in the report the following month. A promise that the promiser controls and the reader can check is the only kind worth writing down.

The first report proposes a target, and the instinct upward is to propose an ambitious one. The service-reliability literature's advice on choosing targets is short and it is aimed at exactly this moment.

• "Don't pick based on current performance." This cuts both ways. Not the team's best night, which is 99.9 percent and which broke as a promise on night 4. And not a round number the room likes the sound of. The target comes from the history, 685 of 720, and from what the reader can absorb, which the settlement lead answered in session 1.
• "Better to start with a loose target that you tighten." 95 percent this quarter is kept, with 3 percent of the budget to spare. It is a promise the team can make and keep in its first quarter of making promises, which is worth more than an ambitious one it breaks in week two. 97 is written down as the target to earn, with the debt item that gets there: the coverage check from session 5, worth 12 partitions, which on its own would have kept the 97 percent budget alive past night 15.
• "Have as few SLOs as possible." One objective on one pipeline for one reader. Not one per indicator, not one per dashboard. The moment there are six objectives on the page it needs a meeting again.

99 is not proposed because the history says the budget would have run out on night 9, and a first report that proposes a target already broken by its own evidence teaches everyone who reads it that the numbers on the page are aspirations. Every report after that is read as one.

The number on the page moves every month. The target next to it moves only at the quarterly review, and the rule is absolute because the two failure modes it prevents are both common. The first is a target quietly loosened in a bad month so that the page stays green. The second is a target tightened in a good month by somebody senior who liked the number, on a pipeline whose history does not support it. Both are the same mistake: a target moved on one month's mood rather than a quarter's evidence.

• The review has the last three reports in front of it. Three months of the same three numbers is enough to see a trend and not enough to forget the incidents behind it.
• The target may move in either direction. Up, if the budget has been kept three months running and the debt that made the difference is done. Down, if three months of history say the promise was never supportable, and it is said out loud that it is moving down and why.
• The reader is in the room. The person who signs the settlement report decides what she needs. The team decides what it can promise. The target is where those two meet, and neither side moves it alone.

The stop rule was set in session 2: when the error budget is burning, features stop and hardening ships. It is easy to agree to in a quiet month. It is tested once, and only once, in a specific way: the budget runs out on night 15, the debt list says build the coverage check, and a feature somebody senior has promised a customer is due on the 20th. Everyone in the room knows what the rule says. The question is who says it.

That is the leader's job in the report, and it is the only part of the whole contract that cannot be delegated. The engineers can measure, design, alert, write the runbooks and the postmortems, price the debt. They cannot tell the person who promised the customer that the feature waits, because they do not have the standing, and if they try, the rule dies quietly and the next report has a footnote explaining why the budget was exceeded. The leader protects the rule by saying, in the meeting, before being asked: the budget is gone, the plan says hardening, the feature moves a week, and here is the debt item that ships instead and what it is worth.

There is a category of product that pages people, tracks incidents, hosts status pages and produces on-call analytics, and a team running the report above will be sold one. The category is real and the products in it are useful. What they are useful for is routing: getting the page to the right person, recording when it was acknowledged, and counting. What they cannot supply is any of the three numbers, because the numbers come from the contract the team wrote: the target from history, the pages from an alert policy the team designed, the toil from a tally the on-call kept.

• Buy the routing, own the numbers. A tool that tells you pages per week is reading your alert policy back to you. If the policy is page-on-every-bad-partition, the tool will faithfully report 8.2 a week.
• The tool's "reliability score" is the single score again, with the denominator owned by a vendor instead of the team. The same refusal applies for the same reasons.
• No comparison between products is offered here, because the report is the same whichever one carries the page, and the choice belongs to the team that gets woken up.