On-call is a design, not a rota

A rota answers one question: whose week is it. It cannot answer what that person will be woken for, how many times, or what they are supposed to do about it. Those answers live in three other places: the alert policy, the load it produces, and the runbook the on-call opens at three in the morning. When the rota is filled in first, the other three are decided by default, which means decided by whoever configured the monitoring, months ago, without anybody's week in mind.

Harborlight's rota was tidy. Six engineers, one week each, handover on Monday. What sat underneath it was an alert on every bad partition, no escalation beyond "call the platform lead", and a runbook that consisted of the previous on-call's chat history. The tidy rota was distributing an untidy design across six people in turn, and each of them believed the problem was theirs.

• Page - an alert that is allowed to interrupt a person, at any hour. Everything else is a ticket, a dashboard or a morning message.
• Alert policy - the rule that decides when a page is sent. Paging on every bad partition is one policy. Paging when the error budget is burning faster than a threshold, measured over a fast window and a slow window, is another. Builder session 3 writes both; the leader chooses between them.
• Load - pages per person per week, counted from history rather than guessed.
• Actionable - a page that was the first word of a real incident, so something the on-call did or said changed the outcome. Defined mechanically on the bench as the first page inside an incident.
• Escalation path - who the on-call is allowed to wake, in what order, written down before the night it is needed.
• Page budget - the number of pages per person per week the team has agreed is acceptable, reviewed the way an error budget is reviewed: exceeded means something changes.
• Business-hours scope - the set of alerts that may only fire during the working day because nothing that matters at night depends on them.

Every page that reaches a person was allowed through by a rule. At most data teams the rule is inherited: the monitoring tool's default, or a threshold somebody set during an incident two years ago and never revisited. "Page on every bad partition" is what that inheritance usually amounts to, and it feels responsible, because it never misses anything. It also never stops.

The alternative the service-reliability practice settled on is to page on the rate the error budget is being spent, over a fast window and a slow window, on the rising edge, with suppression so the same incident does not page twice. The builder track writes the arithmetic. What a leader needs to hold is the shape of the outcome: the second policy pages once per incident, at its first bad hour, and then trusts the person it woke to handle the rest.

Load is pages per person per week, and it is a number in last month's history rather than an opinion. Harborlight's thirty nights, five incidents, two policies:

The last column is the one to read first. Both policies caught every incident. The first policy did not buy any coverage with its extra thirty pages; it bought thirty interruptions about failures the on-call was already awake for. Night 9 alone paged fourteen times for one outage. The fourteenth page carried no information the first had not.

A page without a runbook is a request for improvisation at three in the morning. The runbook is the third input to the rota because it decides how long each page costs and how much the outcome depends on who happened to be holding the pager. Builder session 4 writes them so a tired person can follow the first three steps without thinking; the leader's question is simpler and comes first: for each thing that is allowed to page, does a written answer to "now what" exist at all.

At Harborlight the honest audit was two runbooks for five classes of page, and one of the two was a link to a dashboard. The late load on night 4 had a procedure. The corrupt batch on night 15 did not, and the on-call spent two hours discovering by hand what the settlement team's spreadsheet would have told them in five minutes.

The instinct when a rota is painful is to add people to it. More names, shorter weeks, a second person on nights. Under the first policy that spreads 35 pages across more people; it does not change the 35, and it does not change that 30 of them were noise. Six people each carrying a bad week is not better than three; it is the same fault with a wider blast radius, and the extra people now also believe on-call is miserable.

Fixing the design changes the number itself. The same five incidents, the same five catches, 5 pages instead of 35. That is not achievable by staffing at any headcount, and it is achievable by policy in an afternoon. The order of operations for a leader is therefore fixed: policy, then load, then runbook, and only then the calendar.

"Actionable" is a word that gets argued about, and the argument is usually a way of avoiding the number. The mechanical definition that settles it: a page is actionable if it was the first page inside a real incident. The person it woke learned something they did not know and could act on. Every subsequent page about the same incident is, by definition, telling somebody who is already working on it that it is still happening.

• It is deliberately strict. A page that arrives while the on-call is mid-fix is not useless in every sense, but it is not the reason anything got fixed, and counting it as useful hides the noise.
• It survives disagreement about severity. Every bad partition counts the same. Whether night 21's eight scattered failures mattered to the settlement report is a target conversation, not a paging one.
• It gives a percentage the team can be shown. 14 percent is a number a leader can put in front of the people carrying the pager and say: this is what we have been asking of you.

Four properties, each of which is a decision a leader can make this week without writing a line of configuration.

• One primary, one escalation. One person holds the pager. One named person is the one they wake when stuck. Not a group, not "the team channel", one name per week, and both names are on the calendar.
• Business hours only for anything the settlement report does not depend on. If a failure will not change what the reader sees at nine, it does not need a human at three. It needs a ticket that is read at nine. The test is the reader's sentence, not the engineer's anxiety.
• A page budget per person per week. A number, agreed with the people carrying it, above which the team stops and asks what is wrong with the design. At Harborlight, two pages a week was the number the team said it could carry indefinitely; the first policy was delivering four times that.
• Reviewed like the error budget. Every week the pages are counted against the budget, in the same meeting where the error budget is read. Exceeded means a design action, not an apology to the person who had the week.

The word "escalation" gets treated as a failure of the on-call. It is the opposite: a working escalation path is what stops a hard night from becoming a story about one person's stamina. The path is written down before it is needed. Who to wake first, on what condition, and what to hand them. Below what threshold of time or budget the on-call is expected to escalate rather than permitted to. What the escalation contact is expected to do that the primary could not: authority to call the source team, authority to declare the report late, access the primary does not have.

Harborlight's path before the redesign was a single line: "call the platform lead". After it, the primary was expected to escalate at forty minutes without a fix, or at any page involving an upstream source, and the escalation contact's job was explicitly to own communication with the settlement lead so the primary could keep working. Nobody worked less hard. Fewer people worked alone.

On night 9 the upstream source stopped publishing at two in the morning and did not resume until after six. Fourteen partitions failed. Nothing the on-call could do would fix it; the data did not exist yet on Harborlight's side of the boundary. The right response was to confirm the outage, tell the settlement lead before eight which hours would be missing and when the backfill would land, and go back to sleep. That is a fifteen-minute job.

Under the first policy the on-call was paged fourteen times for it, once an hour, each page for a partition they already knew would fail. The policy punished the person for the source's failure. Under the burn-rate policy the same night produced one page at the first bad hour. The on-call did the communication, silenced nothing, and was not woken again by an outage they had already reported.

This is an observation about teams rather than a number from the bench, and it holds across every team that has run a noisy pager for long enough. People stop reading it. Not out of negligence: as a rational adaptation to a signal that is wrong six times in seven. The on-call learns to glance at the alert title, recognise the shape, and go back to sleep. Then one night the shape is the same and the incident is not, and the one page in seven that mattered is treated like the six that did not.

A noisy pager is therefore not merely tiring. It trains the team to ignore the instrument the whole design depends on. The people carrying it are usually the last to say so, because admitting they have stopped reading the pager sounds like a confession. It is a finding about the policy, and it is the leader's job to say it first.

Session 2 made the error budget a currency: a number of bad partitions the team may spend, and a rule about what happens when it is gone. The page budget is the same instrument pointed at people instead of partitions. A page budget is a number per person per week that has been agreed in the open, is counted from the paging tool rather than remembered, and has a consequence when exceeded.

• Set it with the people carrying it. A budget imposed from above is a target. A budget the on-calls proposed is a promise the leader has made to them.
• Count actionable pages separately. Two useful pages a week is a busy pipeline. Eight pages with one useful is a broken policy. The budget is on the total; the percentage tells you which fix to reach for.
• Exceeded means a design action. Silence an alert, move a class of failure to business hours, change the policy, fix the thing that flaps. Never "we will be more careful next week", which is a wish rather than an action, and session 4 has more to say about that.
• Read it in the same meeting as the error budget. The two numbers move together. A month that spent the error budget on night 15 and the page budget in week one is one month, not two problems.