A correct loader, a green dashboard, and 511 wrong rows
The five sessions so far have been about a number being wrong for reasons you can see once somebody points at them. Tonight is different in kind: the rows are wrong and there is no observation available inside the warehouse that would tell you so. The loader did not fail, the pipeline did not degrade, and the failure is not in anybody's code. It is in the assumption that an upstream only ever adds.
★ What you walk out with tonight. The damage measured on Bench B: 511 orphaned rows at 5.75 percent of the warehouse, stable at 5.65 to 5.75 percent across four runs. A mechanical account of why four standard checks all stay green, including the one that moves in the reassuring direction. The full lag table, 0 through 12, with the dangerous middle named: a lag of 1 leaves 3.99 percent wrong and feels like a precaution. And the parameter the entire table is conditional on, which is not a column in your data and which most teams have never written down.
LiveWithdrawn is not late, and your toolkit only knows late4 min▶
Every warehouse has machinery for records that turn up after the period they belong to: idempotent loads, merge keys, restatement windows. That craft is general, it is well developed, and it belongs to learn-data-engineering-with-phoebe rather than here. What matters tonight is that it is machinery for a different event.
Those patterns handle data that arrives late. This is data that is withdrawn. A late row is additive: something that was missing appears, a total goes up, and nothing that was already correct stops being correct. A withdrawn row is subtractive: something that was counted is gone, a total goes down, and a figure that was right last week can no longer be reproduced. Reaching for a merge key here is not a partial fix, it is a fix for the other case.
Larkfield's engineer asked the reasonable question first: is this not just late-arriving data with extra steps? The team spent two days trying to express it as a restatement window before somebody noticed that every pattern they reached for produced a row, and the thing they needed was to remove one.
Self-studyThe words used tonight4 min▶
- Replacement. The upstream withdrawing a run of recent blocks and continuing from an earlier point. Why it happens belongs to learn-blockchain-with-phoebe; what it does to a table is tonight.
- Depth. How many blocks back a replacement reaches. Bench B fixes it at 3. It is the parameter the whole of tonight turns on.
- Orphaned row. A row already loaded whose block has since been replaced. Still in the table, still queried, wrong, and counted by nobody.
- Finality lag. How far behind the newest block a loader stays before writing a block's rows. Bench B sweeps it from 0 to 12.
- Held rows. Rows that exist and are correct but have not been written yet because the lag has not elapsed. The freshness side of the ledger.
- Append only. A loader that writes and never removes. Correct behaviour, and the thing that turns a replacement into a permanent error.
- Late arriving. A record for an earlier period turning up now. Additive, familiar, and handled by patterns that already exist.
- Withdrawn. A record that was loaded and is now gone from the source. Subtractive, and the case tonight is about.
Four checks, all green, none of them wrong to be
15 min liveFollow five blocks through a loader that appends and nothing else. Three of them are replaced, the replacements are appended too, and the warehouse now holds eight blocks of rows where the chain has five. Then walk the four places a normal pipeline would notice something and watch each of them be satisfied, correctly, by a table that is 3 blocks of rows wrong.
LiveWhy no check fires, one at a time4 min▶
Data quality checks are almost all statements about rows that arrive. Is this row well formed, is the key unique, is the volume in range, is the newest record recent enough. Every one of those is answered by inspecting the incoming batch, and every one of them is answered correctly here, because the incoming batch is fine. The replacement blocks are real blocks and their rows are real rows.
The rows that are wrong are the ones already in the table, and nothing in a normal pipeline goes back to ask whether a row it wrote last week is still supported by the source. That question has no natural home: it is not a schema property, it is not a batch property, and it cannot be answered without re-reading history you have already processed.
The freshness check deserves a moment on its own. After a replacement the newest block is newer than it was, the loader is up to date, and the service level indicator improves. The single observable that moved moved in the reassuring direction, so a team watching a dashboard sees the situation get better at the moment it got worse.
// A plain append-only load. There is nothing wrong with this code.
function onBlock(block) {
warehouse.insert(block.height, block.rows); // append, never update
}
// What a replacement looks like from in here: nothing. The loader is handed
// the new blocks and appends them, exactly as it appends every other block.
//
// There is no error path because there is no error. Rows arrived and were
// written. The rows written earlier were never re-examined, because no
// append-only loader has a reason to re-examine anything.
//
// Note what is missing and cannot be added without a second read of history:
// if (blockWasReplaced(height)) { ... } // nothing supplies this predicate
LiveWhere the wrong rows sit, which is not evenly spread3 min▶
A replacement takes back the most recent blocks, so every orphaned row came from the tip of the chain at the moment the replacement landed. The damage is not scattered uniformly across 200 blocks of history. It is concentrated in short runs immediately before each event, four times over in this run.
That concentration matters for who feels it. Long range aggregates dilute it across a lot of correct history. Anything recency weighted does not: a last-24-hours panel, a daily delta, an alerting rule on the newest window, a cohort defined by when somebody first appeared. Those read disproportionately from exactly the blocks a replacement is most likely to take back.
So the same 5.75 percent produces very different consequences depending on which report you look at, and a team that measures the error only as a share of the whole table will understate what it does to the numbers people actually watch.
511 rows, 5.75 percent, and a table with a hidden parameter
12 min liveRun the undefended loader and read the figure. Then run it again on other draws and watch it stay put, which is what makes it a property of the arrangement rather than an accident of one run. Then look at the lag table, which is the standard first response, and notice two things about it: the middle of it is worse than it looks, and the whole of it is conditional on a number that does not appear anywhere in your warehouse.
...
Live5.75 percent, and it is not a draw4 min▶
At the bench setting the undefended loader ends with 511 orphaned rows, 5.75 percent of the warehouse. Change the run and the share lands between 5.65 and 5.75 percent across four of them. That narrow band is the useful part: this is not a bad day, it is what this arrangement produces, and a team that saw the figure once and hoped it was unlucky will see it again next month.
Now move the depth selector and watch the figure move with it. That is the sentence Part 2 exists for. The share of your warehouse that is wrong is a function of how far back the upstream reaches, and the bench is showing you a family of answers rather than one.
One row of the table is worth pointing at while it is on screen: errors raised, zero. Not zero because the bench is being kind. Zero because there is no code path in an append-only loader that could raise one, which Part 1 walked through line by line.
LiveThe dangerous middle, and why it is worse than nothing4 min▶
Read the first three bars in order. 511, then 346, then 172. A lag of 1 removes about a third of the damage and a lag of 2 removes about two thirds, so both of them are doing something, and that is exactly the problem. The warehouse at lag 1 still holds 346 wrong rows, 3.99 percent of it, and the team now has a documented precaution in place.
A configuration that is wrong and looks unprotected keeps generating the discomfort that eventually makes somebody look at it again. A configuration that is wrong and looks protected does not. The concern has been spent, the ticket is closed, the design document says a finality lag is applied, and nothing on any dashboard will ever contradict that sentence. Partial protection here is not a safe middle position, it is the same error with the alarm removed.
The arithmetic underneath is not subtle. A replacement takes back the last 3 blocks. At a lag of 1 the loader has already written two of those three. At a lag of 2 it has written one. At a lag of 3 it has written none. The sequence 511, 346, 172, 0 is that sentence counted in rows.
LiveWhat tonight deliberately does not answer3 min▶
There are two mechanisms available against this and neither has been described tonight, on purpose. Holding rows back until the source has stopped changing its mind is one. Removing rows already written when the blocks they came from are replaced is the other. They are not two settings of the same thing, they fail in different circumstances, and they send their bills to different people.
Session 7 puts both of them against Bench B and prices them, including the configuration where you build both and the second one never fires. Tonight's contribution to that conversation is the size of the thing being defended against and the question that has to be answered before either defence can be specified.
The useful thing to carry out of the room is therefore not a recommendation. It is a measurement and a question: this much of my warehouse would be wrong, and I do not currently know the depth I am designing against.
Self-studyWhere serious people disagree5 min▶
One camp says 5.75 percent is not an emergency and treating it as one is how engineering time gets burned on the wrong thing. Their case is that plenty of production datasets carry errors of that order from sampling, from clock skew, from a source system's own corrections, and that nobody rebuilds a pipeline over it. They add a sharper point: the error self-limits, because a report run a day later reads mostly correct history, so the practical exposure is confined to the freshest window and can be managed by not making irreversible decisions on the freshest window.
The other camp says the percentage is beside the point, and that quoting it as though it were a tolerance is the actual mistake. Their case rests on three differences. The error is not random, so it does not average out of anything. It is unbounded from inside the warehouse, because the depth could be larger next month and nothing in the table would say so. And it is unmeasured in production: 5.75 percent is a number the bench can produce and your warehouse cannot, so calling it small is a claim about a quantity you have never observed.
Both positions are held by people who ship. Notice that they are not disagreeing about the mechanism or about the 511, and that the first camp's argument depends entirely on its last clause. If your organisation genuinely does not make irreversible decisions on the freshest window, the first position is defensible and cheap. If it does, and most do somewhere, then the exposure is concentrated exactly where the decisions are, and the first argument has been refuted by your own reporting calendar rather than by the second camp.
Find out whether your own upstream can take something back
15 min hands onSix steps on a pipeline in your own estate, chain or no chain. The first three are answerable in ten minutes and most teams have never been asked any of them.
Ask whether your upstream can withdraw
Not restate, not correct, not arrive late. Withdraw: can a record you have already loaded cease to exist at the source with nothing sent to tell you? Write yes, no, or nobody knows. The third answer is the most common and it is a finding.
List the checks you run and mark what each one inspects
Every quality check on the table, with one word each: arriving or resident. Count them. If everything in the list says arriving, you have measured tonight's blind spot in your own pipeline in about five minutes.
Say what your volume alert would do
If your loader wrote a replacement batch alongside the batch it superseded, would any alert fire? Most volume rules trigger on a shortfall. Say out loud what yours does when the count goes up for the wrong reason.
Run the depth selector across its range
In the live box, hold the run fixed and move the depth from 1 to 6. Write down the share of the warehouse that is wrong at each end. That spread is what not knowing the depth costs you in ignorance, expressed as a range you cannot currently narrow.
Write the missing number as a question, not a guess
One line in your design document: the depth this pipeline is designed against is blank, and here is who would know. Resist filling it in from memory. A number in that slot with no source behind it is worse than a blank, because the blank gets asked about.
Name the report that reads the freshest window
One report, dashboard or alert in your estate that leans on the newest data. That is where a withdrawal would land first and hardest. Bring the name to session 7, because it is the thing the two defences are being chosen for.
The habit to keep. Sort your data quality checks into two piles: ones that inspect rows arriving and ones that inspect rows resident. Almost every team has a full pile and an empty pile, and almost nobody has noticed, because the empty pile has no failures in it. A pipeline whose entire quality suite asks about the incoming batch is silent by construction about every way the past can change, and silence from a check that was never written reads exactly like a pass.
Three questions
After a replacement, which of the four checks in Part 1 moves, and in which direction?
Every check inspects rows that arrive, and the rows that arrived are fine. The replacement writes new well formed rows with their own keys, so nothing trips, and the two observables that move both move upward. That is why the situation reads as healthy at the moment it got worse.
A team sets a finality lag of 1 against a depth of 3 and records it in the design document as a precaution. Why is that position worse than having set no lag at all?
A lag of 1 genuinely removes damage, dropping 511 to 346, which is precisely what makes it dangerous. The residual is invisible from inside the warehouse and the documented precaution ensures nobody goes looking for it.
Your team already handles late-arriving records well, with idempotent loads and merge keys. Why does none of that help here?
A merge key resolves a row that turns up after its period. It has nothing to say about a row that was written, counted and reported, and is now unsupported by the source. The first leaves every previously correct figure correct. The second does not.
Homework
Twenty minutes. One inventory and one number, both from a pipeline you actually run.
Finish the arriving-against-resident inventory
Every quality check on one table, sorted into the two piles from step 2, with the count of each. Bring the totals. If the resident pile is empty, write one sentence on what you would have to read twice in order to fill it.
Bring your current lag and where it came from
Whatever delay your loader applies today, in whatever unit, plus an honest note on whether it was a decision or an accident of scheduling. Session 7 prices both defences, and the first question it asks is what you are already paying.