Reading a chain as a dataset

Every engineer has the same opening move on an unfamiliar source and it is not reading the documentation. It is: show me one row, tell me what the columns mean, and tell me what one row is one of. Grain first, because every aggregation you will ever write is wrong if the grain is wrong, and a wrong grain does not raise an error. It quietly doubles a count.

The move works here too, which is the useful news. What changes is the answers. One row is a transaction. The columns are thin and one of them is opaque until you decode it. The grain has two defensible readings rather than one, and which of them you pick determines whether a whole category of analysis is available to you later. None of that requires new engineering instincts. It requires the ordinary instincts applied to a source that will punish skipping the step.

• Address. The only identifier a row carries. Persistent, and not a person. Part 1 tonight, taken apart in session 3.
• Entity. The real party behind one or more addresses. Never present in the data.
• Transaction. One record. It names input addresses, output addresses and a payload, and it sits at a known position in a known block.
• Payload. The part of the record that is meaningless until something decodes it against the definition the deploying code used. Session 2.
• Grain. What one row is one of. Here there are two answers, one per transaction and one per transaction input, and both are used.
• Co-spend. A single transaction spending from two addresses at once. Visible only at the input grain. Session 3.
• Retraction. The upstream withdrawing history you already loaded and reported. Sessions 6 and 7.
• Idempotency, late-arriving data, append-only loads. General pipeline craft, and it all applies. It lives in learn-data-engineering-with-phoebe and is not repeated here.

Two sessions on getting the data into a usable shape, three on counting things that resist being counted, two on the upstream taking history back, and three on what all of that does to a model, a bench and a review.

• 1 and 2. The dataset as it arrives, then the pipeline that turns it into tables anybody can query.
• 3, 4 and 5. The co-spend heuristic and union-find, how to measure what it got right, and what happens when some of the parties are being paid to be hard to count.
• 6 and 7. A chain reorganisation landing in a warehouse that already published, and the two separate defences against it.
• 8, 9 and 10. Training on data where your features and your model are both visible to the people generating the rows, a bench that fooled its own author, and the whole Larkfield estate driven live.

The record is thinner than most sources you will have ingested. There is no event name you did not have to derive, no user column, no timestamp you can trust to mean what a product timestamp means, and no free-text field to profile. In exchange there is total completeness: nothing was sampled, nothing was dropped by a collector, and no agent failed to fire. That combination is rare and it changes where the risk sits. The risk is not missing rows. It is misreading the rows you have.

The one relationship a record does state is the one worth noticing on day one. When a transaction draws on two input addresses, whoever built that transaction had to be able to authorise both. That is a fact about control, stated by the structure of the record rather than inferred from behaviour, and it is the only such fact in the whole dataset. Most teams skate past it because it looks like plumbing. It is the entire basis of every entity count anyone will ever ask you for.

Everything else you might want about a party - who they are, where they are, what they bought, whether they came back - is not thin, it is absent. There is no column with nulls in it. There is no column.

One row per transaction is the natural grain and it is what most people mean when they say a chain is a list of transactions. It is the right grain for counting activity, for time series, for anything where the unit of interest is an event. It has one property worth writing on a wall: the input addresses are a collection inside a single row, so the co-spend relationship exists in your table but is not joinable.

One row per transaction input is the exploded grain. The same transaction id now appears on several rows, one per input address, and suddenly the co-spend is a self join rather than an array operation. That is the shape session 3 needs. It is also the shape that produces a wrong transaction count the first time somebody writes count(*) against it, because the row count is now the input count.

Keep both. Neither is a compromise and neither replaces the other. The failure is not choosing wrongly, it is choosing silently, so that six months later nobody in the room can say which grain the table downstream of them is at. Write it into the table name if you have to.

An address is the most key-like thing in the dataset. It is stable over time, it is unique as a string, it never arrives null, it joins cleanly, and every profiling tool you point at it will report a well behaved high-cardinality identifier. A uniqueness test on it passes. A referential integrity test on it passes. Nothing you can automate will tell you anything is wrong.

What is wrong is the cardinality of the thing behind it. On Larkfield's generated month there are about 1,000 addresses and 304 real parties, and no column, table, vendor or request will move you from the first number to the second. That is a 3.2 times error sitting inside a column that passes every check you own, which is a genuinely unusual failure mode and the reason it survives so long in real reporting.

Sessions 3, 4 and 5 are the response to this. Tonight the only obligation is to stop reading the column as an identity and start reading it as what it is: a handle on a spending capability, one party may hold many.

Standard profiling still works and you should still do it: cardinality per column, null rates, value distributions, the widest and narrowest records, the tails. Point ydata-profiling or whatever you normally reach for at a month of it and read the report the way you always would. The craft does not change.

What changes is the follow-up. Normally a surprise in a profile becomes a question to the owning team, and the answer is either "yes that is a bug" or "no, here is the business reason". Here the surprise is the answer. If the widest transaction has two inputs and you expected more, that is simply what the source does, and there is nobody to confirm it and nobody to change it. Your profile is not a draft to be validated. It is the specification, and you wrote it by looking.

The practical consequence is to profile more often and to keep the profiles. A distributional shift here is not a data quality incident that somebody will explain to you; it is the only warning you will get that the thing producing the rows has changed behaviour.

Take the four absences one at a time and price them honestly, because a session that only lists the costs will be dismissed by anyone who has actually enjoyed working on this data, and they are not wrong to enjoy it.

No SLA is a real loss. Nothing promises you that rows arrive within a window, that the shape stays constant, or that anything at all continues. Your alerting cannot be a breach notification because there is no promise to breach. It has to be a distributional check you wrote against a baseline you measured.

No contract negotiation is a real loss and a small saving. You never spend six weeks agreeing a payload with another team. You also never get to ask for a field, and the field you need is frequently not there.

No backfill request is a gain wearing the clothes of a loss. You cannot ask anyone for history, and you do not need to. The whole history is sitting there and you can re-read it whenever you like, as many times as you like, without anybody's sprint capacity. Re-deriving five years of a metric after finding a bug is a week of compute rather than a quarter of politics. Almost nothing in a normal warehouse works this way.

No gatekeeper is the biggest gain and the one that gets understated. There is no access request, no approval, no scoped credential, no quarterly review, and no possibility of your access being narrowed because a team reorganised. The dependency that most often kills an analysis is the one that does not exist here.

The payload in a transaction is bytes. What those bytes mean is decided by the definition that the deploying code published, and different deployments publish different definitions for what a business would call the same event. You do not get a schema registry handed to you; you get a decision to make about how to interpret each family of records, and that decision is yours, which means it is also your bug when it is wrong.

Then it moves. A new version is deployed with a different definition, and your decode keeps running against the old one. Nothing fails. The rows still parse, or they stop matching and silently fall out of your filter, and either way the first symptom is a metric bending in a direction nobody ordered. There was no deprecation notice because there is no relationship in which a notice could be sent.

This is the single most important operational fact in the whole track and it is why the next session is about the indexer rather than about the chain. The decode is not a technical detail sitting under the pipeline. The decode is the pipeline's opinion about what happened, and it is where every disagreement between two teams' numbers turns out to live.

Look back at the live box. The counts are correct, they were computed rather than asserted, and almost none of them answers a question anybody would ask. You can say how many transactions happened and how many addresses appear. You cannot say how many people, which activity was a reward and which was something else, whether anybody came back, or what any of it means in the product's own vocabulary.

That is not a criticism of the source and it is not a gap to be complained about. It is the normal condition of raw data everywhere. The difference is the size of the distance and who is expected to close it. On a normal source the producing team has already done the first two thirds of that work before you see a row. Here nobody has done any of it, and the whole of it is yours.

Which is the honest case for session 2, so it may as well be said plainly: the thing you are going to build is not a connector. It is an interpretation layer, it will be most of the work, and the numbers it produces will be your numbers rather than the chain's.

One camp says land the raw records first and keep them forever. Their argument is that your decode is an interpretation and interpretations are wrong, so the only reproducible artefact is the undecoded record, byte for byte, exactly as it was read. Keep that and you can re-derive any table from any point with a corrected decoder, and prove to an auditor that the derivation, not the data, was what changed. Throw it away and every historical figure is hostage to a decoder version you no longer have.

The other camp says that is a comforting story that does not survive contact with a real restatement. The raw bytes are only meaningful against the definitions in force at the time they were written, so preserving them without also preserving a complete, versioned map of those definitions preserves nothing but the illusion of reproducibility. And the cost is not theoretical: raw records are large, they are queried by nobody, and the reprocessing run that was supposed to justify them is quietly never scheduled. Better, on this view, to store the decoded rows plus the decoder version that produced them, and to treat a decode change as a schema migration with a proper backfill.

Both camps have shipped this. They are not really arguing about storage, they are arguing about which artefact carries the meaning: the bytes or the interpretation. Notice that the second camp's position depends on actually versioning the decoders, which most teams claim to do and few do. Ask the person in front of you which of the two they hold, and then ask them where their decoder versions are stored. The gap between those two answers is the real risk.