A promise with a number

Two people can both say the orders data is reliable and both be telling the truth about different things. The platform lead means the job ran every night and exited cleanly. The settlement lead means the report she signs at nine was right. The finance analyst means the totals matched the bank. None of those is a number, so none of them can be wrong, so the meeting ends where it started.

A number changes the shape of the conversation. Once the pipeline is "95.14 percent of hourly partitions good over the last thirty nights", the platform lead can dispute the counting, the settlement lead can say whether 95 is enough for her, and the analyst can ask which 35 partitions were bad. Those are three productive arguments, and none of them was available while the word was "reliable".

• The settlement lead reads the report at nine, signs it, and sellers are paid against it. Reliable means: every hour of yesterday is in it and the totals reconcile. She is the reader, and the reader's sentence is the one every number has to answer to.
• The platform lead owns the pipeline. Reliable means: the nightly job ran and exited cleanly. This is a fact about the job, not about the report, and the two came apart ten times last month without anyone noticing.
• The on-call engineer carries the pager. Reliable means: nothing woke me that I could not fix. Sessions 3 and 4 are about this person.
• The finance analyst reconciles the report against the bank. Reliable means: the numbers matched. She is the one who noticed night 15, when every partition landed on time with a currency column that had arrived as text.
• You are the person who has to say a number to all four of them and have it mean the same thing in every direction. That is the job tonight gives you the words for.

The SRE book defines a service level indicator as "a carefully defined quantitative measure of some aspect of the level of service that is provided". For a data pipeline the indicator is a ratio: good events over total events, in a window. The care goes into deciding what one event is and what makes it bad, because every number downstream is arithmetic on that decision.

Harborlight's event is one hourly partition of orders, and a partition is good when it landed within its hour, its totals reconcile, and all of it arrived. That gives 720 events a month, which is enough resolution to see a night that landed 23 of 24 partitions as a partial failure rather than a success. Counting nightly runs instead gives 30 events, and a run that landed 23 of 24 looks exactly like one that landed all of them.

A service level objective is "a target value or range of values for a service level that is measured by an SLI". It is a sentence of the form: this proportion of partitions will be good over this window. It is internal; the team sets it and the team owns it. It is also the thing most teams believe they have and do not.

Harborlight had no objective written anywhere. What it had was a shared belief, "basically 99 percent", which had never been a target because nobody had ever chosen it, and had never been checked because there was no measurement to check it against. A belief cannot be missed. An objective can, and that is its entire value.

A service level agreement is "an explicit or implicit contract with your users that includes consequences of meeting (or missing) the SLOs they contain". The word that matters is consequences. The SRE book gives the test directly: "if there is no explicit consequence, then you are almost certainly looking at an SLO". Money, a credit, a penalty clause, a commitment that costs something to break. No consequence, no agreement.

Harborlight's settlement report has a real one. Sellers are paid against it on a date they were promised, and a late or wrong report moves that date. Nobody in the data team had ever written that down as the consequence of the orders pipeline missing its hours, which is how a pipeline delivering 95 percent had been carrying an implicit agreement nobody had costed.

The instinct in the room, once there is a number, is to ask for all of it. The SRE book is blunt about this: "it's both unrealistic and undesirable to insist that SLOs will be met 100% of the time". Unrealistic because the pipeline depends on an upstream source, a network and a warehouse, none of which is perfect, and a promise of 100 percent is a promise about all of them. Undesirable because the last fraction of a percent is bought with the engineering time that would otherwise ship the changes the business asked for, and because a reader cannot tell the difference between a report that was right every day and one that was right on all but one.

What the literature proposes instead is "an error budget - a rate at which the SLOs can be missed". Set the target at 95 percent and you have said, in advance, that 5 percent of partitions may be bad this month without anyone having broken a promise. That budget is the currency the whole next session is about. Tonight it is enough to know that a promise of less than 100 is not a lowering of standards. It is the first honest statement of what the standard is.

Three numbers were in circulation at Harborlight for the same thirty nights, and nobody had put them in one table.

The orchestrator's 27 of 30 is the one that misled, and it misled in a specific direction. On night 15 the job exited cleanly six times while landing a currency column as text: six partitions on time and wrong. On night 27 it exited cleanly four times while landing partitions with one of three shards missing: four partitions on time and incomplete. Ten bad partitions the success flag counted as good, because a success flag measures whether the job finished, not whether the reader got what she needed. Those ten are most of the distance between the belief and the measurement.

Reliability targets are spoken in nines. The SRE book frames 99 percent as "2 nines" and 99.999 percent as "5 nines", and describes Google Compute Engine's published 99.95 percent as "three and a half nines". The shorthand is useful because it makes the cost visible: every additional nine allows ten times fewer bad events in the same window.

• 99 percent over 720 partitions allows 7.2 bad partitions a month. Harborlight had 35.
• 99.9 percent allows 0.72, which is less than one partition. A single late hour breaks the month.
• 95 percent allows 36, and is not a number of nines at all, which is part of why teams are embarrassed to say it and reach for 99 instead. The embarrassment is not a reason. The history is.

"Is the data reliable" is a question the data team has to answer, and cannot, because it has no unit. "Is 95.14 percent of hourly partitions good enough for the settlement report" is a question the settlement lead can answer in one meeting, because she knows what a missing hour costs her and the data team does not. The number moves the decision to the person with the information to make it.

It also moves the argument. Instead of the platform lead defending the pipeline and the settlement lead defending her frustration, both are looking at 35 bad partitions and asking which of them mattered. Night 9's fourteen missing hours mattered enormously. Night 21's eight scattered hours, each backfilled within the hour, may not have mattered to her at all. That conversation, about which failures the reader can absorb, is the one that sets the target, and it was impossible while the answer was "basically 99".

An agreement can fail in two ways, and organisations tend to have one of each. The first is an SLA with no consequence: freshness by 08:00 written into a data contract, missed a dozen times a quarter, followed each time by an apology. That is an SLO with a grander name, and calling it an agreement erodes every real agreement in the building. The second is an SLA with a consequence you cannot afford: a penalty clause signed at 99.5 percent by someone who never saw the partition log, on a pipeline that delivers 95. That is a liability with a monthly renewal.

• Before signing anything external, know the measured number for the last ninety days. An agreement is a bet on history.
• Keep the internal objective tighter than the external agreement. The gap is the room you need to notice a bad month before it becomes a breached contract.
• Write the consequence down, even internally. "Sellers are paid a day late" is a consequence. Once it is written next to the target, nobody argues that reliability is a technical concern.

Promising a number above the measured one does not make the pipeline better. It makes the promise break earlier. Replaying Harborlight's actual month against four candidate targets shows exactly when.

The pipeline is identical in all four rows. The only thing that changes is how many nights the team spends in breach of its own word, and how quickly the word stops meaning anything. 99.9 percent is what a typical good night at Harborlight actually delivers, and promising it for every night is a promise the history says will be broken before the first week is out.

The SRE book's advice on choosing targets is short and every line of it is aimed at a leader rather than an engineer.

• Don't pick based on current performance. A typical good night is not a target; it is a coincidence. The 99.9 percent row above is what happens when a team promises its best day for every day.
• Keep it simple. One ratio, one window, one sentence. A target that needs a footnote will be misquoted in the first steering meeting.
• Avoid absolutes. "Always", "never" and "100 percent" are not targets. They are the absence of a target dressed as ambition.
• Have as few SLOs as possible. Every objective is a promise somebody has to keep and somebody has to check. Three is a contract. Thirty is a dashboard nobody reads.
• Perfection can wait. It is "better to start with a loose target that you tighten" than to promise a strict one and quietly stop reporting against it in month two.