Why this deep dive exists
When a pipeline breaks, the clock starts. Every minute of confusion is a minute of wrong data, a paged engineer, or a missed SLA. Google's SRE teams treat the doc that ends that confusion - the runbook - as infrastructure, not paperwork. This deep dive upgrades Pulse's Session 5 how-to into a real one.
Docs under pressure 10 min
A how-to guide assumes a calm, competent reader working through a task by choice. A runbook assumes a stressed reader who was just woken up. Same Diátaxis quadrant, stricter rules.
Self-studyRunbooks are infrastructure3 min▶
Google's SRE writing makes a blunt economic case: a good runbook lowers MTTR - mean time to recovery - by handing the responder a proven path instead of asking them to reinvent one under stress. The doc is part of the system's reliability, the same way monitoring and alerting are.
- The 2am reader is your primary audience. This is Session 1's audience statement made concrete: they know your infra but not this specific failure, and they need to act, not learn.
- Every alert should link to its runbook. An alert that fires with no linked recovery doc is half an alert - it announces a problem and abandons the responder.
- A stale runbook is worse than none. A wrong command at 2am costs more than an honest blank. Runbooks get reviewed on a schedule, exactly like the delete-dead-docs rule from Session 7.
Self-studyA runbook is a how-to with stricter rules3 min▶
From Session 5 you know a how-to guide: prerequisites, actionable steps, no teaching. A runbook keeps all of that and adds three constraints the pressure demands:
- Exact commands, not descriptions. "Check the raw folder" becomes
ls -la pulse/data/raw. The reader copies and pastes; they do not translate. - Verify after every action. Each step says how to confirm it worked before moving on. This is the defining trait - more on it below.
- Decision rules, not options. A how-to can say "you might also...". A runbook says "if the count is 0, go to step 4; otherwise go to step 5". No branching left to judgment.
Anatomy of a runbook 15 min
The canonical structure is stable across SRE teams. Read it top to bottom the way a responder does: what fired, what I need, what to do, how to know it worked, what if it did not, who to call.
Self-studyThe eight parts4 min▶
Every reliable runbook carries these, in order:
| Part | What it answers |
|---|---|
| Title + unique ID + short description | What is this and can I search for it |
| Scope, owner, last-reviewed date | Who owns it, is it still trustworthy |
| Trigger / when to use | Which alert or symptom brings me here |
| Prerequisites | What access and tools I need first |
| Steps with exact commands | What do I run, precisely |
| Verification after each step | How do I know that worked |
| Troubleshooting | What if a step does not go as expected |
| Escalation contacts | Who do I page when I am out of steps |
Self-studyWhy the verify line matters3 min▶
The verify line is the runbook's whole reason to exist. Under stress, people skip ahead and assume a step worked. A runbook that says "re-run the pipeline" and stops has failed; one that says "re-run, then confirm daily_metrics.csv shows the day with events above 50" catches the case where the re-run silently changed nothing.
The restart that fixed nothing. A responder followed a runbook that said "restart the ingest service" and closed the incident. The service restarted fine, but the upstream feed was still down, so no data flowed. There was no verify line, so nobody checked. The alert re-fired 40 minutes later. One line - "confirm new rows arrive within 5 minutes" - would have caught it the first time.
Build the Pulse LOW-alert runbook ★ 30 min · build your own
Session 5 gave you docs/how-to-diagnose-low-day.md - a solid how-to. Now upgrade it into docs/runbook-low-alert.md by adding the metadata, exact commands, a verify line per step, a troubleshooting table, and escalation.
Add the metadata block: owner, last-reviewed date, and the trigger - the alert column reads LOW in daily_metrics.csv.
Turn each diagnostic into an exact command with a verify line. "Check raw files arrived" becomes a command plus what a healthy result looks like.
Write the troubleshooting table: three common causes, each with what it means and where it points.
Add the escalation line: who to page when the steps run out, and with what evidence in hand.
Try it yourself ◐ 30-45 min
- Pick one alert your team actually gets paged for. Write its runbook using the eight-part skeleton. If you cannot name the exact commands, that gap IS the runbook's value.
- Audit an existing runbook for missing verify lines. Every "do X" with no "confirm X worked" is a restart-that-fixed-nothing waiting to happen.
- Add a last-reviewed date to your team's runbooks and a calendar reminder to re-verify them. A runbook with no review date is a runbook nobody trusts.
- Optional depth: the ACM Queue article "Why SRE Documents Matter" and the Google SRE workbook cover runbook practice at scale.
Official sources covered
This deep dive combines SRE documentation practice with the Diátaxis how-to model, applied to Pulse. Organization-specific incident tooling stays with each team.
Three questions before you go 🎯 ◐ 90 seconds
1 · What upgrades an ordinary how-to into a runbook?
Pressure demands precision: copy-paste commands, confirmation after every action, and no branching left to a tired reader's judgment.
2 · An alert fires with no linked runbook. What's the problem?
Every alert should link its runbook. Without one, the responder starts from zero at the worst possible time.
3 · Why is a stale runbook worse than no runbook?
A confident but outdated instruction sends the responder down a broken path. Review on a schedule, like deleting dead docs.