The indexer is the product

Most of what follows is not new. Reading an immutable upstream, writing idempotent loads, handling records that arrive out of order, keeping a table restatable, testing a transformation: all of that is ordinary craft, all of it applies here unchanged, and it lives in learn-data-engineering-with-phoebe rather than being repeated tonight. If you already build pipelines you are most of the way there, which is worth saying because the surrounding conversation tends to imply otherwise.

What is left after subtracting the ordinary craft is small, and it is the whole of the difficulty. One stage with no equivalent in a normal pipeline, and three properties of the upstream that break specific habits. Tonight is that residue, stated as precisely as possible so it can be designed for rather than discovered.

• Indexer. The layer that turns raw chain records into tables somebody can query. Four stages, and the subject of tonight.
• Decode. Turning a payload into a named event with typed fields, against the definition the deploying code published. The only stage with no ordinary equivalent.
• Normalise. Folding several record shapes into the one table your team agreed on.
• Enrich. Attaching what you know that the chain does not, from your own systems, clearly marked as yours.
• Aggregate. Counts, cohorts, time series, and the figure somebody puts in a pack.
• Retraction. The upstream withdrawing blocks whose rows you already loaded and reported. Measured in session 6.
• Definition drift. A new contract deployed with a different shape, silently making your decode wrong.
• Idempotency, late-arriving data, restatement. General pipeline craft, applied here unchanged. It lives in learn-data-engineering-with-phoebe.

Decode takes a payload and produces a named event with typed fields. It needs the definition that the deploying code published, and it needs a decision about which family of records you are even looking for. That is the unfamiliar stage.

Normalise folds several record shapes into the one table your team agreed on. It is the same work as reconciling three source systems into a conformed dimension, and the same failure modes apply.

Enrich attaches what you know and the chain does not: your own product's mapping of an address to a campaign, your own labels, your own categories. This is the only stage where private data enters, and every field it adds should be marked as yours rather than the chain's, because a reader six months later will assume every column came from the source.

Aggregate produces the counts and cohorts. It is completely ordinary, and it is where everybody looks first when a number is wrong, which is almost never where the number went wrong.

Say the four out loud in order a few times. The value is not the taxonomy, it is that when a figure is disputed you can ask which stage the dispute is in, and the answer is nearly always stage one.

Look again at the four questions in the amber box. Not one of them has a technically correct answer. Whether an uncollected reward happened, whether an internal movement is a reward, whether a batch of forty is one event or forty, whether a failed attempt is an event at all: these are questions about what Larkfield means, and the decoder is where the answer is recorded. Usually by one engineer, usually under time pressure, usually without anyone calling it policy.

This is why the decode deserves review by the people who own the definition rather than only by the people who own the code. A pull request that changes a threshold in an aggregation is an engineering change. A pull request that changes which payloads are recognised as a reward is a change to what the company reports, and it will move the number without any dashboard showing a deploy.

The practical move is small and it is worth insisting on: write the four answers down in prose, keep them beside the decoder, version them together, and put the prose in front of whoever will be defending the number. If that document does not exist, the definition still exists. It is just only readable by someone who can read the decoder.

Enrichment is the least discussed stage and it carries a specific risk here that it does not carry elsewhere. The whole claimed advantage of working from a public ledger is that a stranger can recompute your figure from the same inputs. The moment stage three attaches a column from your own systems, that property is gone for every metric downstream of that column, and nothing in the table says so.

The fix is naming rather than architecture. Prefix enriched columns, put them in a separate table, or mark them in the model documentation, but make it impossible to read a wide table and not know which columns a stranger could reproduce. Teams that skip this end up publishing figures that look publicly verifiable and are not, which is worse than publishing a figure that is clearly internal.

It is worth being explicit about how much of this is ordinary, because a session that dwells only on the strange parts leaves people thinking they need a new discipline. They do not.

Idempotent loads, watermarking, late-arriving records, incremental models, tests on uniqueness and referential integrity, a restatable published layer, documented lineage, contract tests between your own layers: all of that is exactly what you already do, all of it applies, and none of it changes shape because the upstream happens to be a ledger. That craft lives in learn-data-engineering-with-phoebe and there is no version of it that is special here.

Two habits transfer with a small twist. First, your uniqueness tests should be written against the grain you chose in session 1, and the grain should be in the table name, because both grains will exist in the warehouse. Second, your freshness checks cannot be written against a promise, because there is no promise. They have to be written against a baseline you measured, which makes them a little weaker and considerably more honest.

Every pipeline handles data that is late, data that is wrong and data that gets restated. The habit that covers all three is the same: accept that the past can change, make the load idempotent, keep the published layer restatable. That habit is necessary here and it is not sufficient, because it assumes somebody tells you.

What happens here is that blocks you already read get replaced, and their rows stay in your tables describing history the chain no longer has. There is no restatement message, no correction record, no changed row, no failed load and no alert. The rows are simply wrong from a certain moment onwards and nothing in your stack has any reason to think so.

Session 6 measures exactly how much of a warehouse ends up in that state on Larkfield's numbers, and it is more than most people guess. Session 7 takes apart the two defences, which are separate mechanisms that are constantly confused for each other. Tonight the only thing to carry is that this belongs in the design document as a named property of the source, not as an operational surprise.

Run the live box through all five months and read the spread rather than the values. Four definitions of one metric, every one of them defensible in a review, and the widest and narrowest are hundreds apart on a population of about a thousand addresses. Nobody made an error. There is no data quality incident to open.

This is the part that surprises people who expect a public dataset to produce a single agreed truth. Shared rows remove the argument about what the data says. They do not touch the argument about what the question means, and that argument turns out to be the bigger one. The chain gives everybody the same inputs and then leaves the definition entirely to you, which means the only way two teams agree is by agreeing on words first.

The operational consequence is worth stating flatly. When your figure does not match another team's figure, do not start by comparing totals, and do not start by looking for a bug. Start by putting the two decoders and the two written definitions side by side. The difference is nearly always visible in the first ten minutes, and it is nearly always a sentence rather than a line of code.

One camp says every team must own its own decode. Their argument is that the decode is the business definition, so outsourcing it means outsourcing what your company means by a customer, a reward and an event. A team that adopts somebody else's interpretation inherits every judgement inside it, cannot explain its own number under questioning, and discovers the inherited assumptions only when a regulator or a board member asks the one question the borrowed definition answers differently. On this view the nine weeks Larkfield spent are not waste, they are the cost of knowing what you are reporting.

The other camp says that is an expensive way to produce a number nobody else can compare with. Every team writing its own decode means every team's figure is incomparable with every other team's, which destroys the single genuine advantage of a shared public dataset. Better, on this view, to adopt a published shared definition, contribute corrections to it in the open, and accept that your number will sometimes be wrong in the same way everybody else's is. A figure that is comparable and slightly wrong is more useful than a figure that is bespoke and slightly right, because comparability is the thing you cannot manufacture later.

Both have shipped this and neither is being naive. Notice that they disagree about what the number is for: one camp is optimising for defending it internally, the other for comparing it externally. Notice also that the second position quietly depends on the shared definition being maintained by somebody, and the first quietly depends on your organisation actually writing its judgements down. Ask which of those two conditions holds where you work, and the argument usually answers itself.