Postmortems that change something

A postmortem depends entirely on people telling you what actually happened: what they saw, what they assumed, which alert they dismissed and why. That information exists in exactly one place, the memory of the person who was there, and they hand it over voluntarily. The moment a document says "the on-call failed to notice", every engineer who reads it learns the price of being noticed. Next time, the account arrives later, shorter, and with the awkward part left out. The document was accurate once and made every future document less so.

Blameless is therefore not kindness and not a culture initiative. It is the condition under which the information the postmortem needs continues to arrive. The rule is mechanical: describe what the system did and what the people knew at the time, never what a person should have done. "The alert fired at 02:10 and its title matched the previous six false pages" is blameless and more useful than "the on-call ignored the alert", because it points at a fix. The second sentence points at a person, and people are not a system change.

Thirty nights, 720 partitions, 35 bad. Every number tonight comes from these five rows, and the budget column assumes the 97 percent target from session 2, which allows 21.6 bad partitions a month.

The shares add to more than one budget because the month spent more than one budget. Night 15 is where 21.6 ran out, and every partition after it was a promise already broken.

The test is one question asked of the finished document: what is now different about the pipeline, the alert, the runbook or the contract with the source, such that the same incident either cannot happen or costs less when it does. If the answer is a sentence about the system, the postmortem worked. If the answer is a sentence about people trying harder, the document is an incident report with a moral attached, and the incident will be back.

The narrative is not useless; it is how the room reconstructs what happened. But the narrative is the input, not the output, and postmortems fail when the story is so satisfying that the room mistakes having understood the night for having changed the next one.

A timeline is a list of times and facts, and every postmortem has one. What most lack is the line that converts the facts into the currency the leader track has been building since session 2. Night 9 was not "a bad night" or "a major outage". It cost 14 partitions, which against a 97 percent target and its budget of 21.6 is about two thirds of the month's allowance in one night. That one line does three things a narrative cannot.

• It ranks the incident without an argument. Fourteen partitions against three for the late load on night 4. The room does not need to agree about how bad it felt.
• It tells the reader what was left. After night 9, about a third of the budget remained for twenty-one more nights. Night 15's six partitions spent it. That is why the settlement lead was told the promise was broken in the middle of the month, not at the end.
• It sets the size of the action. An incident that spent two thirds of the budget earns a change with a real owner. An incident that spent a fortieth earns a line in a backlog. The budget decides which, and it decides it before anybody has to defend a feeling.

"Root cause" is a comfortable phrase because it promises there is one thing to fix. There almost never is. Night 9 had a root cause, the source stopped publishing, and it was entirely outside Harborlight's control. A postmortem that stops there concludes "nothing to do", which is false. The night was as expensive as it was because several conditions were true at once: no dependency SLO with the source, so the outage was not the source team's problem until nine; no path to page the source, so the on-call could only wait; a policy that paged fourteen times for a failure nobody on this side could fix; and a settlement lead who learned about it at 08:50 from the report itself.

Remove any one of those conditions and the night costs less. That is the test for whether something belongs in the list: not "did it cause the incident" but "if this had been otherwise, would the incident have been cheaper". Four conditions produce four candidate actions. One root cause produced none.

The last section of every postmortem is a list called actions, and most of what is in it is wishes. The difference is structural, not stylistic: an action names a change to the system, a person who owns making it, and a date by which it is done. A wish names a behaviour and hopes.

The good line can be checked on the 21st. Either the check exists or it does not, and if it does not, the conversation is with Devi about what blocked it, which is a useful conversation. The wish cannot be checked on any date by anybody, which is what makes it comfortable to write and worthless to have written.

• Strike every personal name in the narrative. Replace with a role and what that role knew at the time: "the on-call, holding a pager that had fired six times with the same title". Names survive only in the owner column of the actions, where they are a commitment rather than an accusation.
• Strike "should have". It is always a person and always hindsight. Rewrite as the condition that made the better choice invisible at the time.
• Strike "human error" as a cause. It is a description of the fact that a person was involved. The condition that made the error easy to make is the cause, and it is fixable.
• Keep every timestamp. The gap between the first bad hour and the first human awareness is the detection cost; the gap between awareness and the reader being told is the communication cost. Both become actions.
• Keep what the reader experienced, in the reader's words. The settlement lead's "I found out from the report" is worth more than any internal metric, and it is the line that justifies the communication runbook.
• Keep the budget line. Partitions spent, share of the month's allowance, what remained. It is the sentence that gets quoted upward in session 6.

Three partitions landed after their hour and were backfilled by ten. The reader saw three hours missing at nine and had them by the time she signed. The conditions: the alert fired three times for one delay, and the on-call learned of the delay from the first page and nothing from the next two. The change is a freshness alert on the burn-rate policy that pages at the first late hour and then holds, which is builder session 3's work. About 14 percent of a 97 percent budget, an action with a one-line scope.

Fourteen partitions, two thirds of the budget, and not one of them fixable from Harborlight's side. The fix was never here, which is exactly why this postmortem is the one most teams write as "nothing to do". Two conditions were on this side and both become actions.

• A communication runbook. When an upstream outage is confirmed, the settlement lead is told before 08:00 which hours are missing and when the backfill will land. She learned it at 08:50 from the report. Owner the on-call lead, date the 20th.
• A dependency SLO with the source team. A written expectation of the source's freshness, a path to page them once when it is missed, and the outage on their budget rather than only on Harborlight's. Owner the platform lead, date the 30th, because it involves another team and a conversation.

Neither action makes the next outage impossible. Both make it cheaper: fifteen minutes of communication instead of a morning of surprise, and a source team that knows before nine.

Six partitions landed on time with a currency column that had arrived as text. Every freshness signal said perfect. The report was wrong for six hours and the finance analyst caught it against the bank file. The budget of 21.6 ran out on this night, so this is the postmortem written in the knowledge that the month's promise was already broken.

The change is almost embarrassing in its size: the control total that caught the fault was already being computed by hand in a spreadsheet every morning by the settlement team. Turning it into a correctness check on the pipeline, so a partition whose totals do not reconcile is marked bad and pages once, was one column and one comparison. The condition that mattered was not a missing capability. It was that the check lived on the reader's side of the boundary and the pipeline never saw it.

Eight partitions, scattered across the night, each briefly missing and each backfilled within the hour by the connector reconnecting on its own. The reader may not have noticed at all. The on-call was paged eight times. Two conditions: the connector retried immediately and failed immediately, so each flap was a full failure, and each flap was a separate page. Two actions: retry with backoff on the connector, owner the platform engineer who owns it, and a slow-window alert so eight scattered failures burn the budget visibly and page once rather than eight times.

Four partitions landed on time, with totals that reconciled for the rows present, and each missing one of three shards. Freshness passed. Correctness, as written, passed. The job exited cleanly four times. The reader saw a report that was a third light for four hours. The single condition: nothing measured whether a partition had processed the amount of data it should have. The action is the one already written as a line in Part 1: a coverage check on shard count, a partition with fewer than three shards marked bad and paged once, owner Devi, by the 20th.

Each postmortem above is correct on its own. The finding none of them produces appears only when the five are read as a set: nights 21 and 27 are the same gap. A flapping connector that drops a partition and a load that lands two shards of three are both failures of coverage, the indicator that asks whether a partition processed as much data as it should have, and Harborlight had no coverage indicator at all. Two incidents, twelve partitions, more than half of a 97 percent budget, one missing measurement.

• A single postmortem cannot see it. Night 21's document is about a connector. Night 27's is about shards. The word coverage appears in neither because the people writing each were, correctly, looking at their own night.
• The review is a standing meeting, not a reaction. Monthly, all postmortems since the last one, read for repeated conditions rather than repeated incidents. Repeated incidents are obvious. Repeated conditions hide behind different names.
• It changes the size of the action. Two small actions, a retry and a shard check, become one indicator that would have counted both nights and any future night that looks like neither. That is the change the review earns, and it is the one that gets reported upward in session 6 as "the month found a missing indicator", which is a sentence a steering committee can act on.