Where session 7 sits
The contract has a target, a budget, alerts, a runbook, a backfill and an incident package. Session 7 adds the third number the leader report carries: hours of toil per person per week, next to budget remaining and pages per week. Session 8 then replays the month on the bench and shows what the alert policy did to the first two of those numbers. Toil is the one that decides whether anybody is still on the rota in a year.
★ What you walk out with tonight. A six-property test that tells toil from overhead and from engineering, so the same task is classified the same way by everyone on the team. Last month's toil at Harborlight as a number: 35.1 hours, 4.1 per person per week on a rota of two. A toil budget agreed the way an error budget is. And the ordering rule that says the repeat pages go first, then the manual backfills, which is exactly what burn-rate alerting and the idempotent job already did.
LiveWhy toil is the number that predicts whether the rota survives3 min▶
Incidents are visible. Everybody remembers night 9. Toil is invisible because it arrives in twenty-minute pieces: a page acknowledged, a query pasted, a backfill started and watched. None of the pieces is worth a ticket, so none is counted, and a team can lose two days a week to work nobody would list if asked. The consequence is circular. The team that has the most toil has the least time to remove it, so toil grows until somebody leaves, and the person who leaves is the one who carried the most of it.
Measuring toil breaks the circle by making it a number that can be over budget. Once it is over budget, removing it is not a favour the team asks for; it is the contract.
Harborlight's two on-call engineers each believed toil took "an hour or two a week". The on-call log and the ticket queue said 35.1 hours in the month, which is 4.1 hours per person per week, with 15 of the 35.1 spent acknowledging the 30 threshold pages that only repeated what the first page of each incident had already said. Nobody had lied; nobody had counted.
Self-studyThree kinds of work, and why the boundary matters3 min▶
- Toil - operational work that is manual, repetitive, automatable, reactive, leaves nothing lasting behind, and grows with the number of pipelines. Acknowledging the thirteenth repeat page of night 9's outage.
- Overhead - work that keeps the team running and is not tied to the pipeline: standups, planning, reviews, hiring. Necessary, not toil, and not counted against the toil budget, because automating a standup is not the fix for a heavy rota.
- Engineering - work that removes toil or adds lasting value: the burn-rate alert that replaced 35 pages with 5, the idempotent backfill job, the runbook rewrite. Manual, yes; repetitive, no.
The boundary matters because a team that logs its runbook rewrite as toil will report a toil number that goes up in the month it fixes things, and a team that logs its repeat pages as "monitoring" will report a toil number of zero while drowning.
Toil is manual, repetitive, automatable, and grows with the estate
9 min liveToil is defined by its properties, not by how unpleasant it feels. A task is toil when all six hold at once. One or two is not enough: engineering is manual too, and a standup is repetitive.
LiveThe six properties, and the two neighbours toil gets confused with3 min▶
A task is toil when it is manual (a person does it), repetitive (the same task recurs), automatable (a machine could do it as well), reactive (it interrupts rather than being scheduled), leaves no lasting value (the system is in the same state after as before), and scales with the number of pipelines rather than with the business. That last property is the sharp one for a data team: ten more pipelines means ten times the backfills by hand, with no more revenue to show for it.
Overhead fails "automatable" and "scales with pipelines": planning does not double when the estate does. Engineering fails "repetitive" and "no lasting value": the runbook is rewritten once and the rewrite is what stops the next twenty-nine minutes of improvisation. Both are manual. Manual alone tells you nothing.
LiveHarborlight's toil last month, enumerated honestly4 min▶
Honest means including the small things, because the small things are where toil hides. The on-call log and the ticket queue for last month gave six items:
| Toil item | Occurrences | Minutes each | Hours in the month |
|---|---|---|---|
| Repeat threshold pages acknowledged (35 pages, 30 of them repeats) | 30 | 30 | 15.0 |
| Backfills run by hand, watched to completion | 3 | 180 | 9.0 |
| Stale-partition tickets from analysts, answered by hand | 12 | 20 | 4.0 |
| Weekly manual check that the settlement export ran | 4 | 45 | 3.0 |
| Runbook step: copy the diagnosis query out of the wiki, edit the hour | 5 | 25 | 2.1 |
| Month-end reconciliation with finance, by hand | 1 | 120 | 2.0 |
Total: 35.1 hours in the month. Two people carry the rota, so 17.6 hours each, which over 4.3 weeks is 4.1 hours per person per week. The 30 repeat pages are the bench's 35 threshold pages minus the 5 that were the first word of an incident: 14 percent of the pages were actionable and the other 86 percent were toil at two in the morning.
Self-studyWhat was left out, and why it matters that it was3 min▶
- Sleep lost is not in the table. Thirty repeat pages at night cost far more than 30 minutes each in the next day's output, but that cost is not in any log, so it is stated in words next to the number rather than invented as a number.
- The incident work itself (diagnosis, safe state, coordination) is not toil. It is reactive and manual, but it is not repetitive in the toil sense and it produces the incident package. It is counted separately as incident hours.
- Standups and planning are overhead, logged but not counted. A team can hide toil by calling it overhead, which is why the classification is written down per category, not decided per entry.
- Work by people not on the rota is counted if it is toil: the analyst who re-runs a dashboard by hand every Monday because the partition is late is doing the pipeline's toil for it.
A toil budget, and what to automate first
8 min liveA measured toil number is only useful with a ceiling next to it. The ceiling is agreed the way the error budget is: above it, engineering time goes to removing toil before it goes to features, and that rule is written into the contract so it does not have to be re-argued every sprint.
LiveMeasuring it: hours per person per week, from the logs you already have3 min▶
The unit is hours of toil per person per week, because that is the unit people feel and the unit a sprint is planned in. The sources are the on-call log, which records every page with its acknowledgement and close, and the ticket queue, which records every request a human answered by hand. Each entry gets a category, each category is classified once as toil, overhead or engineering, and the hours are summed. Estimating durations is allowed when the log has none, as long as the estimate is written down and the same for every occurrence of that category.
The honest version counts the twenty-minute pieces. A team that only counts the backfills will report 9 hours and believe it; the repeat pages were 15.
LiveThe toil budget, agreed like the error budget3 min▶
The error budget says the pipeline may have 21.6 bad partitions at 97 percent, and above that, features stop until reliability work has been done. The toil budget says the same about hours: Harborlight agreed 3 hours per person per week as the ceiling, and last month's 4.1 was over it. Being over is not a judgement about the team; it is the trigger. The next sprint's engineering time went to burn-rate alerting and the idempotent backfill job before anything else, and the toil fell to 1.3 hours per person per week, under the ceiling, with room for the estate to grow before the trigger fires again.
Without the written ceiling, the same two pieces of work had been proposed for three quarters and lost to features every time, because "the pager is a bit noisy" is an opinion and a feature deadline is a date. With the ceiling in the contract, 4.1 against 3 was a breach, and a breach gets the sprint.
Self-studyThe ordering rule: frequency times duration3 min▶
Automate the item whose occurrences times minutes is largest. Not the most annoying, not the one somebody already has a script for, and not the one that is easiest, because easy items are usually small. At Harborlight the rule produces this order:
- Repeat pages, 15.0 hours. Removed by multi-window burn-rate alerting, which pages once per incident: 35 pages to 5, 8.2 a week to 1.2. Session 3's work was the largest toil reduction of the year and nobody had called it that.
- Backfills by hand, 9.0 hours. Removed by the idempotent backfill job from session 5, which is re-run from a watermark and watched by nobody.
- Stale-partition tickets, 4.0 hours. Next. Publishing the partition status the indicator query already computes lets analysts see for themselves that the 09:00 partition is late, instead of asking.
- Re-rank after each removal. The wiki query is copied once per incident, so it stays at 2.1 hours; the export check stays at 3.0. Orders change as items are removed, and the list is recomputed monthly rather than remembered.
Self-studyToil worth keeping, and writing that decision down2 min▶
Not every item on the list should be automated. The month-end reconciliation with finance is two hours once a month: 24 hours a year. Automating it means matching finance's spreadsheet logic, which changes each quarter, and would cost about a week of engineering plus maintenance. Doing it by hand is cheaper for now, and the decision is written down with the number that justified it and a date to revisit. What is not acceptable is leaving it unclassified, because an unclassified item is toil that has been made invisible again.
Measure a month of it
★ 20 min · everyone buildsBring your on-call log and your ticket queue for the last thirty days. If you have neither, Harborlight's ops.oncall_log has 35 pages, 12 tickets, 3 manual backfills and the rest of last month in it.
Pull the log and classify each entry. Every entry gets a category; every category is mapped once to toil, overhead or engineering. Anything unmapped shows up as its own row so it cannot hide.
Compute hours of toil per person per week. Sum the toil hours per person, divide by the weeks in the window. Harborlight: 35.1 hours, two people, 4.1 each per week.
Rank toil items by occurrences times minutes. The top of the list is what to automate first. Print the whole list, including the two-hour items.
Mark what sessions 3 and 5 already removed. Repeat pages under burn-rate alerting, backfills under the idempotent job. Subtract them and re-rank what is left.
Write the toil budget you would agree and the first automation you would fund. A ceiling in hours per person per week, the current number against it, and the top remaining item with its hours.
Write the three numbers for the leader report. Budget remaining at the current target, pages per week, toil hours per person per week. One row, dated, appended monthly.
The budget number does not move between the two rows above, and that is correct. Burn-rate alerting and the idempotent job changed the pages and the toil; 35 bad partitions still happened, and at 97 percent the budget was still gone on night 15. The lever for that number is session 2's target and the engineering the budget breach buys, not the alert policy.
Before session 8
40 min- Classify every entry in your own on-call log for the last thirty days. Write down the category-to-kind mapping as a table, not in the query, so the next person classifies the same way.
- Compute hours of toil per person per week and compare with what each person on the rota guessed before seeing it. Keep both numbers.
- Rank by occurrences times minutes and identify the top item. Estimate the engineering cost of removing it and compare with a year of doing it.
- Propose a toil ceiling in hours per person per week to whoever approves the sprint, with last month's number next to it. Bring the response.
Three questions
Which of these is engineering rather than toil?
The rewrite is manual, but it is not repetitive and it leaves lasting value: it is the work that removes toil. A and B are all six properties at once.
Harborlight's toil list has repeat pages at 15.0 hours, manual backfills at 9.0, and a month-end reconciliation at 2.0. What does the ordering rule say to automate first?
Frequency times duration ranks the hours, and the hours are what the rota is losing. The repeat pages were the largest toil reduction available and session 3 had already made it.
Why is the month-end reconciliation left as manual toil, and what makes that acceptable?
Some toil is worth keeping for a while. What is not acceptable is leaving it unclassified, because that makes it invisible again.