Backfills and debt as reliability work

Ask any data team to show you its reliability backlog and it will have one: a column, a label, a list of twenty things everybody agrees should be done. Ask when the oldest item was written and the answer is usually a year or more. The list does not shrink because nothing on it ever competes with anything. Features are argued for in a planning meeting with a customer's name attached. Incidents are argued for by the pager at three in the morning. The reliability list is argued for by nobody, so it waits for a quiet month, and there is no quiet month.

There are two kinds of work on that list, and they behave differently. Backfills are the work an incident leaves behind: partitions that were bad or missing and have to be recomputed so that the history reads true. They are urgent and invisible at the same time, so they get done, badly, by whoever is on call. Debt is the work that would have prevented the incident: the check that was never written, the retry that was never added. It is important and never urgent, so it does not get done at all. Tonight prices both in the same unit the error budget uses, because that is the only unit in which they can be compared with a feature.

The postmortems in session 4 explained the five incidents. What they did not do is list the work each one left behind, because that work was already being done by the time the postmortem was written, by the on-call, unrecorded.

• Night 4, late load, 3 bad partitions. The three partitions landed, late. Nothing to recompute. What was owed was a message to the settlement lead before eight saying which hours were late, and whether the report she signed at nine included them.
• Night 9, upstream outage, 14 bad partitions. Fourteen hours never arrived. Once the upstream came back, 14 partitions had to be recomputed, and every report that had been read with 14 hours stale needed a conversation with whoever read it.
• Night 15, corrupt batch, 6 bad partitions. Six partitions on time with a currency column that arrived as text. A backfill: six partitions recomputed with the fix, and six hours of a signed settlement report that now read differently.
• Night 21, flapping connector, 8 scattered bad partitions. Eight small backfills, each run by hand within the hour by the on-call. None of them appeared on any plan, and the on-call's evening appeared on no timesheet.
• Night 27, partial load, 4 bad partitions. Four partitions landed on time and correct with one of three shards missing. A backfill: four partitions recomputed with the third shard, plus every report that had consumed them at two thirds.

Five incidents, five different kinds of owed work, all of it done by two people between midnight and nine over the course of a month, none of it planned, none of it priced.

A pipeline writes one partition per hour. When a partition is bad, because it arrived late, or wrong, or with a piece missing, or never arrived at all, the history has a hole or a lie in it at that hour. A backfill goes back to that hour and computes the partition again, from source data that is now available or now correct, and writes it over the bad one. When it is finished, somebody reading last month sees what should have been there, not what was.

Three words in that description carry the whole cost. Again means compute that was already spent once. Over means anything that read the old partition read something that no longer exists. Last month means somebody may already have acted on the old version. The builder track owns how to run a backfill without breaking the reports downstream, with idempotent writes and watermarks and replay. A leader owns whether it is funded, who is told, and when it is allowed to run.

A plan has two kinds of line on it. Features, which somebody asked for and which have a name and a date. Incidents, which nobody asked for but which announce themselves loudly enough to get a line after the fact. A backfill is neither. It is not a feature, because nobody outside the team asked for last month's history to be correct; they assumed it was. It is not an incident, because the incident is over by the time the backfill starts. So it has no line, and work with no line lands on whoever is on call, in the hours after the page, unrecorded.

• It has no requester. The settlement lead did not ask for night 15 to be recomputed. She asked for the report to be right, and assumed the recompute was part of the pipeline being a pipeline.
• It has no date. A feature slips and somebody notices. A backfill that has not been run yet is a history that is quietly wrong, and nobody notices until a reader reconciles against it.
• It has no owner. The on-call did it because the on-call was awake. When the rota turns, the half-finished backfill belongs to nobody.

Priced honestly, a backfill has three costs, and they are paid by three different people.

The first cost is the only one a data team can pay alone, which is why it is the only one most plans see. The second grows with every hour the fix waits, because every hour another report reads the wrong partition. The third does not grow; it is either there or it is not, and when it is there it changes what kind of work the backfill is.

Five incidents, 35 bad partitions, and three quite different kinds of after-work. Putting them side by side is what shows that "backfill" is not one thing.

Night 9 is the one that looks like the biggest backfill and is really the biggest conversation: 14 partitions is an afternoon of compute, but 14 hours of a report that people read and acted on is a list of people to call. Night 15 is the smallest recompute and the heaviest, because the number it changes had a signature on it. Night 21 cost nothing anybody could see and one person's whole night.

A backfill that changes a number nobody has signed is a pipeline task: schedule it, run it, tell the downstream owners. A backfill that changes a number somebody has signed is a different thing, and the difference is not technical. The signature meant that a person took responsibility for that number and other people acted on it: sellers were paid, a month was closed, a figure went into a board pack. Silently correcting it afterwards means the record of what was decided no longer matches what was decided on. That is the definition of a governance problem, and it is handled as one.

• It is announced before it runs, never discovered after. The person who signed the number hears, from the data team, what is about to change, by how much, and why, with the old and new figures side by side.
• The signer decides what happens to the signed version. Restate it, footnote it, leave it and correct forward. That is their call, because their name is on it. The data team's job is to make the call possible, not to make it.
• The old version is kept. A backfill overwrites the partition. It does not overwrite the report that was produced from it, because that report is now a record of a decision.
• The announcement is written down. Who was told, when, what changed. When the auditor asks in nine months why March's settlement differs from the March that was paid, that note is the answer.

The announcement is short, arrives before the run, and gives the signer everything they need to decide what to do about the signed version. It is not an apology and not a technical explanation.

Every sentence in the second version answers a question the signer will have. Which hours, why, when, by how much, in whose favour, what happens to what I signed, and what do you need from me. The numbers in it are the actual ones the recompute will produce, run in a scratch table first, because "the numbers might look a bit different" is the sentence that turns a governance event into a governance failure.

Debt in a codebase is a shortcut somebody took. Reliability debt is slightly different: it is a thing that was never built, because the pipeline worked without it, until the night it did not. Three kinds recur in every data team, and each one maps directly onto an incident that has not happened yet.

• The check nobody wrote. A coverage indicator that compares a partition's row count and shard count with the same hour last week takes a morning to build. Without it, a partition that lands on time and correct with a third of its data missing scores as good, and the reader finds out at reconciliation.
• The runbook that stops at step two. Step one: acknowledge the page. Step two: check the orchestrator. Step three: blank, because whoever wrote it knew what to do next and did not write it down. At three in the morning, that blank is the difference between a fifteen-minute fix and a fourteen-partition outage.
• The connector with no retry. A source that drops the connection once an hour for thirty seconds produces eight scattered bad partitions a night, each one a page, each one fixed by hand. A retry with a back-off is a day's work. The eight pages are a month's.

None of these is an incident. Each of them is the reason an incident will happen, and the incident will be priced in bad partitions when it does. The debt can be priced the same way now, before the date is filled in.

Session 2 set up the error budget as a currency: at a 97 percent target, 21.6 bad partitions a month may be spent without a promise being broken. When the budget is healthy the team ships features. When the budget is burning, the same currency buys something else: hardening, the work that stops the burn. That is the conversation reliability debt belongs in, because the debt list is the shopping list for a burning budget. It answers the question the budget conversation always reaches: we are burning, what do we buy?

• When the budget is healthy, the debt list waits, honestly and by decision, not by neglect. It is reviewed at the monthly report and nothing on it is funded unless it is cheaper than the incident it prevents.
• When the budget is burning, features stop and the top of the debt list ships. Not the whole list: the item whose incident is the one currently burning the budget.
• When the budget has run out, as it did on night 15 at a 97 percent target, the rest of the month is hardening by definition. The plan does not need a meeting to decide this. The stop rule decided it in advance, and session 6 is about who protects that rule when a feature is late.

The effect on the debt list is that it stops being a list of chores and becomes a list of purchases with prices attached, waiting for the budget to say which one to make. A list like that shrinks.

The five incidents in Harborlight's month look, in the postmortems, like five different stories. The late load was upstream timing. The outage was upstream being down. The corrupt batch was a type that drifted. The flapping connector was a network that dropped once an hour. The partial load was a shard that never arrived. Five causes, five action items, five owners.

Two of them share something the postmortems did not make obvious, because postmortems are written one incident at a time. Night 21's flapping connector produced eight partitions each missing a slice of the hour. Night 27's partial load produced four partitions each missing one of three shards. Both are the same failure from the reader's point of view: a partition that landed on time, reconciled, and was not all there. Both would have been caught by one coverage indicator comparing row count and shard count against the same hour last week. That check had been on the reliability list for seven months.

Twelve of 35 bad partitions is a third of everything the month spent. At a 97 percent target with a budget of 21.6, those 12 are more than half the budget. One item, one morning of work, and it would have been the difference between the budget running out on night 15 and the budget lasting the month. That is what "worth" means for a debt item, and it is a number the budget conversation can use directly.

The instinct, once a team has named its reliability debt, is to give it its own home. A separate board, a separate column, a "tech health" epic. It feels like taking the work seriously. It is the opposite: it is the mechanism by which the work is guaranteed never to happen. Anything on a separate list is by definition not on the list that gets planned, so it never has to beat a feature, so it never gets to.

• One plan, one list. The coverage check sits in the same ordered list as the new dashboard and the new export, and it is either above them or below them. Its position is a decision somebody made and can be asked about.
• Priced in the same unit as the thing it competes with. A feature is justified by what it earns or saves. A debt item is justified by the bad partitions it prevents, at the budget's exchange rate. "Twelve bad partitions, more than half a month's budget at 97" is a sentence that can sit in a planning meeting next to a customer's name.
• Reviewed when the budget is reviewed. Not in its own ceremony. When the monthly report says how much budget was spent and on what, the debt list is the second page: what would have prevented it, and what it costs.

A debt item is priced twice: what it costs to do, and what it would have cost not to do. The first is an estimate in days. The second is a count of bad partitions from history, and it is the one that makes the argument.

• Find the incidents it would have prevented. Read the last quarter's incidents and ask, for each one, whether this item would have caught it before the reader did. Be strict; "might have helped" does not count.
• Sum their bad partitions. Nights 21 and 27 at Harborlight: 8 and 4, 12 in total. That is the item's worth in the budget's currency.
• State it against the budget. Twelve of 35 bad partitions is a third of the month's spend. Against a 97 percent budget of 21.6 it is more than half. Against a 95 percent budget of 36 it is a third. Say which target you are pricing against, because the same 12 partitions are a different fraction of each.
• State what it displaces. One morning of an engineer's time, and whichever feature that morning was going to.
• Write the one sentence. "Build the coverage check before the export: it would have caught 12 of last month's 35 bad partitions, more than half the budget at our target, for one morning of work." That sentence goes in the plan, above the export, and it is the whole argument.