Runbooks a tired person can follow

The person who writes a runbook knows the pipeline, is awake, and is sitting at a keyboard. The person who reads it is none of those things. Night pages at Harborlight went to a rota of six engineers, four of whom had never touched the orders pipeline, and the median page arrived at 02:40. Every design decision on the page follows from writing for that reader: short, exact, and free of anything that asks them to weigh options.

A runbook that says "check whether the source looks healthy" has asked a tired stranger to make a judgement about a system they do not know. A runbook that says "run Q1; the expected output is one row with answered = 0" has asked them to compare two things. The second one works at 02:40. The first one produces a different outcome every night, depending on who is on the rota.

• Runbook - one page, bound to one alert, that takes the reader from the page to a safe state. Not a troubleshooting guide, not an architecture document.
• Safe state - the condition of the downstream report that the runbook aims for: held, served stale, or scheduled for backfill. Decided per report, in advance, by the people who read that report.
• The first three steps - confirm the page is real by looking at one thing, decide whether the report is at risk by one rule, tell one named person. The same on every runbook, before any diagnosis.
• The line - the point on the page below which exactness is required and above which no decision is permitted. Everything above the line is a lookup or a comparison.
• Confirming query - the one query that proves the page is the incident this runbook covers, with its expected output printed under it so the reader compares rather than interprets.
• Binding - the alert definition names its runbook; the page carries the link. A page without a link is a page asking the reader to remember where the document lives.
• Drill - running the runbook against a past night's history with the clock pinned, timed, and recording the date and minutes at the top of the page.

Diagnosis is not the job. Fixing is not the job. The job is that, some minutes after the page, the downstream report is in a state its readers agreed to in advance and the person who owns that report knows. If the reader of the runbook also fixes the pipeline, good; the runbook did not ask them to, and it would still have done its job if they had not.

That framing is what makes a runbook usable by somebody who did not build the pipeline. A stranger cannot diagnose an unfamiliar system at 03:00, and a page that asks them to is a page that gets closed and an incident that gets found by the settlement team in the morning. A stranger can run a query, compare its output to a printed expectation, run one command and send one message. So that is what the page asks.

Every runbook for the orders pipeline opens with the same three steps, in the same order, and none of the three requires a decision. They are the same on all five runbooks so that the on-call learns them once and does them from memory on any page.

• 1 · Confirm the page is real by looking at one thing. Run the confirming query. Compare the output to the expected output printed beneath it. If it matches, this is the incident this runbook covers. If it does not, the page names which runbook to open instead. No interpretation of dashboards, no "does this look normal".
• 2 · Decide whether the settlement report is at risk, by one rule. The report is signed at 07:00 against the previous day's 24 partitions. If the affected partitions belong to a day whose report has not been signed yet, the report is at risk. One query against the sign-off table says which; the reader reads the answer, they do not weigh it.
• 3 · Tell one named person. The settlement duty lead on the rota, by name, before anything else is attempted. The message is one sentence: what is missing, from when, and that the report is going on hold. Diagnosing first is the mistake every tired engineer makes, because diagnosis feels like progress. Telling the reader of the report is progress.

Only below those three does the page say anything specific to the incident. The rule is drawn on the page as a literal line, and the discipline is that nothing above it may ask the reader to choose.

The left column is short because a phone screen is short. The right column is not deleted; it is moved to the document whose reader is awake. A runbook that carries the right column is a runbook the on-call scrolls past looking for the query, and every scroll at 03:00 is a chance to give up.

At 00:00 on night 9 the orders source API stopped answering. Fourteen partitions in a row did not land. The burn-rate page fired at hour 0. There was nothing on the Harborlight side that could make a source API answer, and the pipeline itself was healthy: it was polling a dead endpoint and correctly writing nothing.

A debugging tree for that night has no useful branches. Restarting the connector produces the same nothing. Checking credentials produces the same nothing. The runbook's step 4 for the upstream outage is therefore not a fix; it is a template message to the settlement team saying what is missing, since when, that the report is held with a banner, that nothing on this side can shorten the wait, and when the next update comes. The most valuable thing the on-call can do for fourteen hours is send that message at 00:20 and rerun the confirming query every thirty minutes until the source answers.

The burn-rate policy from session 3 is one query, and it decides whether to page. It does not decide which runbook. That comes from the newest bad row in the partition log and one probe of the source: did the partition land at all, was it late, was it reconciled, did all three shards arrive, and is the source answering. Those fields sort every incident Harborlight has had into one of five shapes, and each shape is an alert with a name.

• The alert definition names the runbook. orders.burn.source-down names runbooks/orders/upstream-outage.md and nothing else. One alert, one runbook. An alert that could mean two runbooks is a decision left for the reader, above the line.
• The page carries the link. The on-call taps the page and the runbook opens. Nobody searches a wiki at 03:00; a runbook that has to be found is a runbook that is not opened.
• The window that fired is not the qualifier. Which window crossed its line says how fast the budget is burning, not what broke. Night 9 and night 21 both paged on the fast window at their first bad hour; the log row is what tells them apart, so the log row picks the name.

Pipelines change every week. Runbooks change when somebody remembers. Six months after it was written, a runbook's confirming query references a column that was renamed, its hold command points at a report that moved, and its named person left in March. Nobody notices, because with five pages a month across five runbooks each one is opened roughly once a month, and the person opening it at 03:00 is the least equipped person in the company to spot that the page is out of date.

The fix is to run it when nothing is wrong. A drill takes a past night from the partition log, pins the clock to an hour inside the incident, and walks the runbook top to bottom exactly as the on-call would, timing it. The confirming query either returns the printed expected output or it does not. The hold command either works against a staging copy of the report or it does not. The named person either answers or has left. Quarterly is the cadence that catches a rename before a real page does, and the date and minutes go at the top of the page, so the next reader knows how fresh the page is before they trust it.

"Safe" means something different for every downstream reader, and the on-call is the wrong person to decide which. A settlement report that is signed and paid against must never show a stale total, so its safe state is hold with a banner. An operations dashboard that people glance at is better stale than blank, so its safe state is serve stale with a banner saying how old the numbers are. A demand forecast retrained nightly can wait a day, so its safe state is backfill later, with no banner and a ticket.

Those three rows were decided in a meeting with the people who read each report, in daylight, once. The runbook's safe-state section is one row of that table copied in, with its command. The on-call runs the command. They do not revisit the meeting.

• The month's shape sets the runbook's shape. Under burn-rate alerting Harborlight's month is 5 pages, one per incident, 1.2 a week. Five runbooks cover it completely, and each is opened about once. That is why they are short: a document read once a month is read by somebody who has forgotten it.
• Threshold paging would have opened them 35 times. Fourteen openings of the outage runbook on night 9 alone, each one after a page saying the same thing. The runbook's design assumes the session 3 policy; under threshold paging the reader would be on step 1 for the fourteenth time at 13:00.
• Once a month is not enough to keep a page true. Hence the drill. A runbook that is used daily corrects itself; a runbook used monthly has to be exercised on purpose.
• Every bad partition weighs the same here, and so does every runbook. The corrupt-batch runbook protects a signed number and the late-load runbook protects an hour of waiting. A more mature set would order the five by the harm they prevent. Start with five of equal weight and let the incident packages from session 6 tell you which to invest in.