The data is public now

Ask a data leader what their team depends on and you get tooling, headcount and a warehouse. The four things that actually hold the work together are further down and rarely said: we own this dataset, we can correct it, we can attach a person to a row, and the only people optimising against our metrics are us.

None of those is a technology choice. They are properties of the arrangement, and they are so reliable that the entire craft is built on top of them without a mention. A public ledger removes all four in one move, which is why the experience of working on one is disorienting rather than merely unfamiliar. It is not that the tools are different. It is that four load-bearing assumptions are gone and the tools were built assuming them.

• Address. The only identifier a row carries. It is not a person and it is not an account you can look up. Part 2 tonight, measured in session 2.
• Entity. The real party behind one or more addresses. The gap between an address count and an entity count is session 2's whole subject.
• Append only. New records can be added. Existing records cannot be altered. Session 4.
• Adversarial data. A dataset where some rows exist because somebody wanted your metric to read differently. Session 3.
• Pseudonymous. Persistent identifiers with no attached identity. Not the same as anonymous, and the difference matters more than it sounds.
• Provenance. The record of where a figure came from and what produced it. The one place a public ledger hands a data team something better than usual. Session 5.
• Record linkage. Joining records that describe the same party through fuzzy attributes and probability. That craft lives in learn-entity-resolution-with-phoebe; the evidence here is structural instead, so the machinery differs even where the problem rhymes.
• Scoping. What to ask before agreeing to build on any of this. Session 6.

The first coupling is the one people expect to hurt most and it is the most double-edged of the four. Larkfield's competitor can query exactly the rows Larkfield queries, at the same moment, with the same completeness. A data moat built on exclusive access to those rows does not exist and cannot be built, because the rows were never Larkfield's to withhold.

That removes an advantage. It also removes an excuse, and the second half is the part that gets left out. When the underlying rows are public, "our numbers say something different" stops being the end of a conversation. Anyone can go and look. A team that has been quietly resting on the unverifiability of its own reporting finds that support gone, and a team that has been reporting carefully finds that its care is now visible to people who could not previously check.

Neither of those is a small change to how a data function is run. The first shifts where the advantage has to come from: not the rows, but what is built on top of them, which is a harder and more honest place to compete. The second shifts what a reported figure is: a claim that can be audited by a stranger rather than a claim that has to be taken.

Appending a fix is possible. Changing a record is not. Read that twice, because the first half makes the second half sound survivable and it is only survivable if every consumer of the data knows the difference.

The usual restatement machinery assumes a rewrite somewhere. A wrong figure is corrected, the corrected figure is what the table now says, and the history of the error lives in a changelog that nobody reads. Here the error is a row, it stays a row, and the fix is a second row. Every reader who does not know to look for the second row is reading the first one and getting the wrong answer, permanently and with no warning.

The general craft of append-only loading and late-arriving data is handled in the data engineering courses. Session 4 deals with the twist that those do not have to handle: the upstream can withdraw history that has already been loaded and reported.

Every row carries an address. An address is persistent, so it behaves like an identifier, which is exactly the trap. It looks like the customer id column that every analyst has used since they started, and it will pass every schema check and every join that column would pass. It just does not mean what that column means.

One person can hold four addresses and often does, for reasons that have nothing to do with concealment. Four addresses can also be one party holding sixty. Nothing in the data distinguishes those cases, and nothing outside the data can be joined in to settle them, because the key that would do the joining does not exist and is not being withheld from you by anyone you could ask.

Session 2 measures what this does to Larkfield's headcount, on generated data where the truth is known and the answer can be checked rather than asserted.

One camp holds that a dataset you cannot correct has no business being the system of record for anything a regulator, a customer or a court might ask about. Their argument is not squeamishness: it is that every governance regime in existence assumes a correction path, so a dataset without one cannot be brought into compliance by effort, only by building a second, private, correctable store beside it and treating the public one as an input. On this view the right answer is usually to keep the ledger out of the reporting line entirely.

The other camp holds that the correction path was always the problem. A dataset that can be quietly rewritten has no provenance worth the name, and the reason nobody noticed is that the rewriting was done by people acting in good faith. An append-only record forces every correction to be visible as a correction, which is the first time most data functions will have had genuine, externally checkable lineage. On this view the discomfort is the machinery working, and the teams complaining loudest are the ones whose previous numbers would not have survived it.

Both positions are held by people who have shipped regulated data products. They point at different first moves: one starts by drawing a boundary and keeping the ledger outside it, the other starts by rebuilding the reporting layer so that a superseded row is a first-class concept. Which one the person across the table holds tells you more about how the project will go than any technical answer they give you.

Change the month and watch the address count move around a thousand. That figure is solid: it is a count of distinct things in the data and no judgement went into it. Then look at the row underneath, which is the row the board actually asked about, and notice that the correct entry is not a worse number. It is a statement that the data does not contain the answer.

A data team's reflex at this point is to go and find the answer, and there is a standard method for doing so. Session 2 runs it on this same data, where the truth is known, and reports how close it gets. The result is not what most people expect, in a direction most people do not guess.

In an ordinary pipeline the people who can move a metric and the people the metric is about are different sets, mostly. Here they overlap completely. Larkfield issues a reward for activity; activity is what the metric counts; creating activity costs one new address. There is no separating layer, because the thing being measured and the thing being paid for are the same event, written to the same row.

The consequence for a data function is that data quality stops being a function of the pipeline and becomes a function of what the product pays for. A metric that nobody has an interest in moving can be fixed by better engineering. A metric that somebody is being paid to move cannot, and treating it as an engineering problem produces a cleaner pipeline reporting the same wrong number. Session 3 is where that gets measured.

Three of the four couplings cost Larkfield something, and it would be easy to leave tonight with the impression that the whole arrangement is a downgrade. That would be the wrong conclusion and it would also be a useless one, because Larkfield is not going to stop shipping the product.

The credit side is real and it is mostly provenance. A figure derived from these rows can be recomputed by anyone who doubts it, from the same inputs, without asking Larkfield for access. Two parties who disagree about a number can settle the disagreement by running the derivation rather than by escalating. A regulator can check a claim without an audit engagement. None of those is available on a private warehouse at any price, because the bottleneck there is not the quality of the data, it is that the other party has to be granted access to it and then has to trust that what they were granted is complete.

Whether that credit outweighs the three costs depends entirely on what is being built, which is why it is session 5 and session 6 rather than a slogan. Session 5 takes the credit side seriously enough to say where it fails too.