Where tonight sits in the builder track
Eight sessions, one artifact: a reliability contract for one pipeline. Sessions 1 to 3 measured, promised and alerted; session 4 wrote the page the on-call follows. Three of Harborlight's five incidents left wrong or missing data in the table after the night was over, and the contract needs a backfill procedure that repairs it without doubling rows and without changing a signed number in silence. Session 6 writes the incident package that records what the backfill changed.
★ What you walk out with tonight. A partition step that is safe to run twice, written as delete-then-insert keyed on partition_hour inside one transaction. A watermark that records how far the pipeline has correctly processed, and the move that sends it back to the first bad partition. A consumers table the backfill job reads so the notification is mechanical. And the sequence for night 15 in the order it has to happen: identify, announce, recompute, re-trigger downstream in dependency order, compare the totals, file the diff.
LiveWhy a rerun is the most dangerous thing a pipeline does3 min▶
A pipeline's normal run touches each partition once. A backfill touches partitions that have already been touched, that downstream jobs have already read, and that people have already made decisions from. Every assumption the normal run gets for free - the partition is empty, nobody has read it, the number can still change - is false during a backfill. The two failures that follow are different in kind.
The first is mechanical: a step that appends rows will append them again, and the partition now has twice the orders. The second is human: a step that replaces rows correctly will correctly replace a total that the finance director signed at 07:00, and nobody will know until the two versions of the report are side by side in a dispute. The first failure is fixed with a delete. The second is fixed with a message sent before the delete.
Harborlight's previous backfill, months before night 15, reran a week of partitions to fix a timezone bug. The step appended. Seven days of orders were doubled, the settlement totals for the week doubled with them, and the first person to notice was a merchant whose payout arrived twice. The recompute was correct; the rerun was not idempotent. Every rule tonight is downstream of that week.
Self-studyThe vocabulary tonight adds3 min▶
- Idempotent - running the step twice leaves the table exactly as running it once would. For a partition step: the partition's output is replaced, never added to.
- Partition overwrite - the idempotent shape: delete the partition's rows, insert the recomputed ones, both keyed on
partition_hour, inside one transaction so a reader never sees the gap. - Watermark - the pipeline's record of how far it has correctly processed. A single timestamp per pipeline; every partition at or before it is trusted.
- Backfill - moving the watermark back to the hour before the first bad partition and running the idempotent step forward to the last bad one.
- Signed number - a total that a named person has approved and others have acted on. It can still be corrected; it cannot be corrected silently.
- Consumers table - a record of which reports, exports and models read which partitions, so the backfill job can list who to tell and what to re-run.
- Replay - re-triggering the consumers of the corrected partitions in dependency order, so a report that feeds an export is rebuilt before the export is.
- Backfill log - one row per backfill: the partitions, the total before, the total after, when it was announced, when it ran, who ran it.
A backfill is a rerun that must not double anything
8 min liveTwo properties make a rerun safe, and they are properties of the pipeline, not of the person running it. The step must replace its own output exactly, so running it twice equals running it once. The watermark must say how far the output is correct, so a backfill knows where to start and does not reprocess a month to fix six hours.
LiveIdempotency: a partition rerun replaces its own output exactly3 min▶
The test is one sentence: run the step twice and the table is identical to running it once. For a partitioned table the only shape that passes is replace the partition: delete every row whose partition_hour matches, insert the recomputed rows for that hour, and do both inside one transaction so no reader ever sees an hour with zero rows. Platforms that offer insert overwrite partition do the same thing in one statement; the guarantee is the same.
Never append during a rerun, and the cleanest way to guarantee that is to never append at all: if the normal nightly step is the same delete-then-insert, a backfill is the normal step pointed at an old hour. One code path, tested every night, is the code path that runs when it matters.
LiveWatermarks: how far the pipeline has correctly processed3 min▶
A watermark is one timestamp per pipeline, processed_through, and its meaning is strict: every partition at or before it is correct. The nightly run reads it, processes the hours after it, and advances it. A backfill is the same run with the watermark first moved back to the hour before the first bad partition. For night 15 that is 04:00: hours 05 to 10 recompute, hours 00 to 04 are never touched, and the six hours after 10 were already right.
- Back to the first bad partition, not to the start. The temptation is to "rerun the month to be safe". Idempotency makes that harmless to the table and expensive to everything else: 720 partitions of compute, 720 partitions of downstream re-reads, and a signed report for every one of the thirty days now changing under its readers. Six hours is six hours.
- The watermark is also a signal. Downstream jobs that process "everything since the watermark" see it move back and know that range is being rewritten. Moving it is how the pipeline tells the truth about what it currently trusts.
- Night 9 needed no reset. Fourteen partitions never landed, so the watermark never advanced past night 8 at 23:00. When the source came back the normal run started from the watermark and caught up. That is the watermark doing its job with nobody touching it.
LiveThree incidents needed a backfill, and each needed something different2 min▶
| Incident | Bad partitions | What was wrong in the table | What the backfill needed |
|---|---|---|---|
| Night 15 · corrupt batch | 6, hours 05 to 10 | Landed on time, currency column parsed as text, amount_base wrong | Recompute the six with the fixed currency parsing, after announcing, because the report was signed against them |
| Night 27 · partial load | 4 | Landed on time with two of three shards | Rerun the four once the missing shard was available; the watermark had advanced past them, so it moves back |
| Night 9 · upstream outage | 14 | Never landed | Recompute the fourteen once the source was back, and tell everybody who read the report stale that the numbers are now real |
Nights 4 and 21 needed none. The late load and the flapping connector delivered their rows late, and the connector's retries completed the short partitions before morning; the table ended those nights whole and only the reader's wait was counted. A backfill is for wrong or missing data, not for a bad indicator.
The downstream has already read the wrong numbers
9 min liveBy the time a backfill runs, the bad partitions have been read. Reports were built from them, exports were sent, a model was trained, and on night 15 a settlement total was signed. Correcting the table is the easy half. The hard half is that every one of those readers now holds a number that is about to be wrong, and some of them acted on it.
LiveThe signed number: a backfill that changes it is a governance event3 min▶
At 07:00 on the morning after night 15 the settlement duty lead signed a total of 1,284,310.55. Six of the partitions behind it had a currency column that arrived as text, and the base-currency amount for those rows was wrong. The recompute produced 1,271,902.10. Both numbers are the output of the same pipeline; only one of them has a signature on it, and merchants were paid against it.
A backfill that silently replaces a signed total has not fixed a bug. It has created two versions of a document that people act on, with no record of which one anybody saw. The rule is that a backfill touching partitions a signed report read is announced before it runs: to whom the report was signed by, with the old total, the expected direction of the change, and the time it will run. Then it runs. Then the old and new totals are written down next to each other, with the announcement time, in a place that outlives the chat thread.
Finance's own spreadsheet caught night 15 at 09:50, when its control total disagreed with the signed report. The question they asked first was not "how do we fix it"; it was "what did we pay against". A backfill log with the signed figure, the corrected figure and the announcement time is the answer to that question, and it is one insert statement.
LiveConsumers are a table the job reads, not a list somebody remembers3 min▶
"Who read these partitions" is a question with a changing answer. A new export is added, a model starts reading the table, a report is retired. If the backfill's notification depends on the engineer remembering the list, the list is wrong within a quarter and the newest consumer is the one nobody tells.
The fix is a consumers table: one row per downstream job that reads the orders table, with its kind, its owner, its dependency rank and the command that re-triggers it, plus a read log that records which partitions it consumed and when. The backfill job joins its target partitions to the read log and gets the list mechanically. On night 15 that list was three rows: the settlement report, the payout export that depends on it, and the demand forecast. Each row carries the person to tell and the order to run in, and the announcement in step 2 is generated from it rather than composed.
LiveThe replay order: partitions, then downstream in dependency order, then compare3 min▶
- Backfill the partitions first. All six, inside the transaction, so no downstream job is re-triggered against a half-corrected range. The partition log rows get a
backfilled_atstamp;reconciledstays false, because the month's record is what the reader saw that morning and the 35 bad partitions stay 35. - Re-trigger downstream in dependency order. Rank 1 before rank 2. The settlement report rebuilds first; the payout export reads the rebuilt report and so runs after it; the forecast reads the table directly and can run alongside the export. Running the export before the report re-sends the wrong payouts with a fresh timestamp.
- Compare the totals last. The signed figure against the figure the rebuilt report now shows, for exactly the same partitions. The diff is the size of what the readers were told wrong, and it goes in the log next to both totals.
- File the diff. One row in the backfill log: partitions, total before, total after, announced at, ran at, who ran it. The incident package in session 6 quotes that row; nobody reconstructs it from memory.
Self-studyThe honest cost: compute, attention and where it gets budgeted2 min▶
- A backfill spends compute. Six partitions recomputed, three downstream jobs re-run, one of them a model. Night 9's fourteen partitions were more than half a night's work redone in daylight, on the same warehouse the daytime analysts use.
- A backfill spends attention. The announcement, the reply from finance, the merchant who asks why the payout moved. None of that is on a dashboard and all of it is real.
- A backfill spends trust, once. A signed number that moves with an announcement is a correction. One that moves without it is the last time that report is trusted without a second spreadsheet beside it.
- The cost belongs in the budget conversation. The error budget from session 3 counts the bad partitions; it does not yet count what repairing them cost. Putting that cost next to the budget, and deciding how much of a quarter's engineering time backfills may consume before the pipeline itself gets fixed, is a leadership decision, and the leader track's session 5 is where it is made.
Backfill night 15 safely
★ 20 min · everyone buildsUse Harborlight's partition log, the orders fact table and the consumers table. The steps run in the order the strip shows: identify, announce, recompute, re-trigger, compare, file. The announcement comes before the recompute on purpose, and the before-total is captured when the announcement goes out, because after the delete it no longer exists anywhere except the signed report.
Identify the bad partitions from the partition log. Night 15, reconciled = false. Put the hours in a targets table so every later step reads the same list. Harborlight: six rows, hours 05 to 10, all on time, all three shards.
List the consumers and announce, capturing the before-total. Join the targets to the read log to get who consumed them, in dependency order. Insert the backfill log row now with total_before and announced_at. Send the announcement generated from those rows. Do not proceed until it has gone.
Write the idempotent recompute. Delete-then-insert keyed on partition_hour, inside one transaction, with the fixed currency parsing. Run it. Count the rows. Run it again. Count again: the same.
Reset the watermark to the hour before the first bad partition. 04:00 for night 15. The scheduler's next tick runs the same step forward from there and re-advances the watermark; tonight you have already run it by hand, and idempotency is why doing both is harmless.
Re-trigger downstream in dependency order. Rank 1, wait, rank 2. The commands come from the consumers table, not from memory.
Compute the settlement total after, compare with the signed figure, and complete the log row. total_after, finished_at, and the diff. The row is what session 6 quotes.
Before-total first, or it is gone. Once step 3's delete runs, the only place the old total exists is the signed PDF. Capturing total_before in the same statement as the announcement is what makes the diff in step 6 a fact rather than a reconstruction.
Before session 6
30 min- Find the step in your own pipeline that writes the fact table and run it twice against a copy of one partition. Count the rows after each run. If the counts differ, rewrite the step as delete-then-insert keyed on the partition before doing anything else tonight.
- Find your pipeline's watermark. If it does not have one, add
processed_throughto a watermarks table and have the run read and advance it. Then move it back one hour by hand and watch what the next run does. - Build the consumers table for your fact table: every report, export and model that reads it, with owner, dependency rank and the re-trigger command. Ask each owner one question: "if these numbers changed, would you want to know before or after". Their answer is the announcement rule for that row.
- Write the backfill log row for the last backfill your team ran, from memory and from whatever records exist. Note every field you could not fill in. Session 6's incident package will ask for the same fields.
Three questions
A partition step is run twice by mistake. Under which shape is the table identical after the second run?
A filtered insert still appends: 4,120 rows becomes 8,240. The transaction stops readers seeing the gap, but only the delete stops the doubling. Idempotency is a property of the step, not of how carefully it is run.
The settlement report for night 15 was signed at 07:00 against six corrupt partitions. The recompute is ready at 11:30. What has to happen before it runs?
Correct and silent creates two versions of a document merchants were paid against, with no record of which one anybody saw. Announce first, then run, then file both totals with the announcement time.
Why does the backfill move the watermark back to 04:00 on night 15 rather than rerunning the month "to be safe"?
Idempotency makes the full rerun harmless to the table and expensive to everything else. Six bad hours is six hours of recompute; the watermark is what lets the backfill know that.