The bench has an answer sheet and your warehouse never will
Session 3 built the method and derived the structural zero. It ran against a world where the generator knows every owner, which is why the scoring function can report 392 still split and 0 wrongly merged. Strip that away and you are left with what a real run gives you: an address count, a merge count and a cluster count. No truth column, no held-out set, no labels arriving later. Tonight is the reporting problem that leaves behind, and it has a good answer rather than a resigned one.
★ What you walk out with tonight. The two directions an entity-resolution error can run, and the derivation of why only one of them is available here. The consequence stated precisely: 696 is an upper bound on entities and there is no lower bound to pair it with, so the honest output is a bound with a direction rather than a point. A second, non-adversarial source of over-count, measured: on month A only 787 of the 982 addresses ever appear as a spend input, so the method has no evidence at all about the other 195. And three checks that still work with no answer sheet, plus a clear statement of what each one does not prove.
LiveWhy accuracy is the wrong thing to ask for4 min▶
The reflex when a method produces a number is to ask how accurate it is, and the reflex is trained on situations where accuracy is measurable: a held-out set, a label that arrives next week, a reconciliation against a system that does know. None of those exists here. There is no second source that holds the mapping from address to holder, because the mapping was never recorded by anybody. It is not withheld, it is absent.
So the request "how accurate is this" has no answer, and the useful move is to stop trying to produce one. What replaces it is not a weaker version of accuracy. It is a different kind of statement: a bound, with a proven direction, that holds for every run on every month regardless of what the truth happens to be. That claim is smaller and it is checkable, which is the trade worth making.
Larkfield's first attempt at this put a confidence interval around the clustered figure. It was computed by bootstrapping the transaction sample, and it was tight and completely meaningless: it measured how much 696 would wobble if a slightly different set of transactions had been drawn, which is a question nobody asked. The distance to 304 is not sampling variation and no resampling scheme can see it.
Self-studyThe words that keep coming back4 min▶
- Still split. Clusters beyond the true entity count. Links the method failed to make. On Bench A it runs 372 to 403 across the five months.
- Wrongly merged. A cluster holding addresses from more than one party. Measured at 0 on every seed, for the reason session 3 derived.
- Upper bound. A number the truth cannot exceed. Here, the cluster count.
- Lower bound. A number the truth cannot fall below. Here, absent, and the absence is the point.
- One-sided error. An error that can only run in one direction. A consequence of evidence that is positive only.
- Ever spent. An address that has appeared at least once as a transaction input. The only addresses the method can say anything about at all.
- Monotonicity. The property that the cluster count can only fall as more data arrives. Follows from addresses minus merges equals clusters.
- Window. The span of history fed to the clustering. The only lever this method has, since it has no threshold and no parameter.
- Record linkage. Fuzzy attribute matching with agreement weights, where a precision and recall trade-off does exist. That craft is learn-entity-resolution-with-phoebe and it is a different evidence base.
Two directions, one of them empty, and what that buys
15 min liveAn entity-resolution method can be wrong twice. It can put two parties in one cluster, and it can leave one party in two clusters. Most methods do both, in a proportion you tune, which is why they come with a threshold and an operating point. This one does exactly one of them, and the column proving it is 0 on all five months of Bench A. The consequence runs all the way to the sentence you are allowed to write in a board pack.
LiveThe two directions, and why one of them is empty4 min▶
Over-merging puts two parties into one cluster and pushes the count below the truth. Under-merging leaves one party spread across several clusters and pushes the count above it. A method that does both has an error that could fall either way, which is why such a method gets reported with an interval and why the interval is honest.
Session 3 derived why the first direction is unavailable: producing the co-spend that would fuse two parties requires authorising both inputs, and a party that can authorise both is one party. There is no data the generator could produce, and no data a ledger could carry, in which that case exists. The measured 0 in the wrongly merged column across five seeds is a confirmation of a derivation, not a finding.
So the error vector has one component and it is non-negative. That is a much stronger statement than a small error. A small two-sided error still needs a caveat in both directions. A one-sided error of any size at all lets you write the word "most" in front of the number and be right about it forever.
LiveA ceiling with no floor is still worth publishing3 min▶
The obvious complaint about a one-sided bound is that it is half a result. The complaint is correct and it is not a reason to withhold it, because the alternative on the table is not a two-sided bound. It is the raw address count, which is also an upper bound, is also one-sided, and sits about 286 further from the truth on month A.
Framing the comparison that way changes the conversation. Both candidate figures are ceilings. One of them is a lower ceiling. Nobody has to argue about accuracy, because neither figure has any, and the choice is between two bounds of the same kind where one dominates.
What you lose by not having a floor is the ability to say the count is not much smaller than this. You cannot say it and you should not imply it, and the imply is where teams actually fail: a figure reported with no qualifier will be read as an estimate by everybody downstream, which is why the word is not optional and the direction has to travel with the number into every place it goes.
// Month A, the seed convention from session 3: world seed s, tx seed s + 1.
var spent = {};
txs.forEach(function (inputs) {
inputs.forEach(function (a) { spent[a] = (spent[a] || 0) + 1; });
});
var everSpent = Object.keys(spent).length;
var report = {
addressesSeen: world.addresses.length, // 982
everSpentAsInput: everSpent, // 787
noEvidenceAtAll: world.addresses.length - everSpent, // 195
merges: cl.links, // 286
entitiesAtMost: cl.entities // 696
};
// There is deliberately no entitiesAtLeast field. Nothing in the run
// produces one, and adding a plausible-looking number here is the
// single most damaging line anybody could write in this file.
Live195 addresses the method never had a word to say about4 min▶
The gap between 696 and 304 gets attributed to operators almost every time it comes up, and part of it is. There is a second contributor that is entirely ordinary and easy to miss, because it is invisible unless you go and count it: on month A only 787 of the 982 addresses ever appear as a transaction input at all. The other 195 are seen only as recipients.
Those 195 cannot be clustered. Not clustered badly, not clustered with weak evidence: union is only ever called on inputs, so an address that is never an input is never an argument to it, and it finishes the run exactly where it started, alone. Each one is a cluster of size one, which means at least 195 of the 696 clusters are addresses the method was never given a single fact about.
That is worth naming separately whenever the gap is explained, because it is not adversarial, it is not fixable by a better heuristic on the same evidence, and it will be a larger share of the gap in any product where most participants receive more often than they send. Larkfield issues rewards, so its data leans that way by design.
Three checks that survive when the truth column is deleted
12 min liveTake away trueEntities and stillSplit and the run still supports real checks. None of them measures accuracy, and saying so plainly is part of running them properly, because a check that gets mistaken for an accuracy measure is worse than no check. What they catch is a broken implementation, a figure that moved for the wrong reason, and two months that are not comparable.
...
LiveRead the production view first, then turn the truth back on4 min▶
Start with the truth hidden, because that is the table you will actually have. Five months, five cluster counts between 676 and 707, five merge counts, and an identity that holds on every row. Everything on that screen is computable from rows anybody can read, and nothing on it tells you how far from correct the figure is.
Now ask the question a reviewer will ask: is 696 a good number? The production view cannot answer it and neither can you, and the correct response is to say what the figure is rather than how good it is. At most 696 entities, computed over this window, by a method whose only error direction is upward.
Then turn the truth on. The two extra columns appear and the gap is 372 to 403 depending on the month. Nothing else on the row changed. That is the whole lesson: the run was identical, the reporting was identical, and the thing you were tempted to promise was never in the run at all.
// 1. Monotonic identity. Cheapest test in the file, catches most build errors.
if (score.addresses - cl.links !== cl.entities) throw new Error("union-find is wrong");
// 2. Monotonicity under more data. Feed a prefix, then the whole thing.
var half = W3.clusterByCoSpend(world, txs.slice(0, txs.length / 2));
if (cl.entities > half.entities) throw new Error("more evidence raised the count");
// 3. Stability across seeds. A smell test, deliberately loose, and not a proof.
// Month A to E land on 676, 688, 696, 696, 707: spread 31 on a figure near 690.
if (spread(runs) / median(runs) > 0.10) console.warn("seed-fragile, go and look");
// Note what is absent: no test compares any of this to a true entity count,
// because outside the bench there is not one to compare to.
LiveThe check that looks rigorous and is not3 min▶
Resampling is the trap, and it is attractive because it produces an interval and intervals look like rigour. Bootstrap the transactions, re-run the clustering, and you get a distribution of cluster counts that is narrow and stable. It is a real measurement of a real thing: how much the figure would move if a slightly different draw of transactions had happened.
That is not the error. The error is 392, it is structural, and every bootstrap sample contains the same 195 never-spent addresses and the same ordinary people who never happened to co-spend. Resampling cannot reach any of it, so the interval it produces is tight around the wrong centre and will be read by everyone as a statement about the distance to the truth.
The same objection applies to cross-validation, to a holdout of transactions, and to splitting the month in half and comparing. All of them measure sampling behaviour on a dataset whose problem is not sampling. If you want one sentence for a review: an interval that does not contain 304 is not an interval about the entity count.
Self-studyWhere serious people disagree5 min▶
One camp says report the bound as the headline figure, with the word "at most" fused to it in every system that carries it, and treat any stripping of that word as an incident. Their case is that a bound is a genuine result, that the direction is derived rather than assumed, and that the alternative is somebody else quoting the raw address count with no qualifier at all. They point out that regulated reporting is full of bounds and that consumers of numbers cope with them fine when the discipline is enforced at the source.
The other camp says that in practice the qualifier does not survive contact with a slide, an email or a quarterly comparison, so publishing a bound is publishing a point estimate with extra steps and a defence you will not be present to make. Their proposal is to report the address count, which is at least trivially defined and impossible to misread as people, and to keep the clustered figure internal as an analysis artefact.
Both have watched the failure they are worried about. The first camp's risk is that the bound loses its qualifier in transit; the second camp's risk is that the worse number becomes the published one and gets the same treatment anyway. Notice that the disagreement is not about the method at all: both sides accept 696, both accept that it is a ceiling, and what they differ on is whether your organisation can carry a qualifier for four quarters. That is a question about your organisation, and you already know the answer.
Write the report you would actually publish
15 min hands onSix steps, most of them with the live box open and the truth hidden. The aim is to leave with a sentence you would put your name on and a test file that fails if anybody later adds a lower bound to it.
Count the never-spent addresses yourself
Build the set of addresses that appear as an input at least once. On month A you should get 787 against 982 addresses. Subtract. Those 195 are singleton clusters before the algorithm does anything, and confirming that in your own code is the fastest way to believe it.
Hide the truth and write the figure
With the toggle on production, write down the one number you would publish for month A and the exact words attached to it. No hedge sentence underneath. The words have to travel with the number, so they have to be short enough to survive.
Try to write the lower bound
Genuinely attempt it. Whatever you reach for, ask what observation in the rows would move it. The exercise ends when you can say out loud why the field cannot be filled, which takes about two minutes and is the point of the step.
Run the identity at three window lengths
Feed a quarter, a half and all of the transactions. Check addresses minus merges equals clusters at each. Check the count never rose. You now have monotonicity as a measured fact rather than a claim from a slide.
Bootstrap it on purpose and look at what you get
Resample the transactions with replacement twenty times and take the spread of cluster counts. It will be small. Write next to it the distance from 696 to 304, and keep both numbers side by side as a reminder of what an interval here is measuring.
Write the test that blocks a future floor
One assertion that fails if the report object ever gains a field naming a minimum entity count, and one comment saying why. The person who adds it in eight months will not have been in this room, and the test is the only part of tonight that will still be there.
The habit to keep. When somebody asks how accurate a figure is, check first whether accuracy is defined for it. If there is no source that holds the truth, the question has no answer and producing one anyway is the failure mode, not the honesty. Say which direction the error runs, say what could move it, and say what you measured instead. That is a complete answer even though it does not contain a percentage.
Three questions
A colleague bootstraps the transactions, re-runs the clustering twenty times, and reports 696 plus or minus 4. What is wrong with that interval?
Resampling can only see variation it can resample away. The 392 gap on month A is a property of what the evidence contains, not of which transactions were drawn, so a tight interval around 696 is a correct measurement of an irrelevant quantity and will be read as a claim about the truth.
On month A, 787 of the 982 addresses ever appear as a spend input. What does that tell you about the 696 clusters?
Union is only ever called on transaction inputs. An address that is never an input is never unioned and finishes the run alone, so each of the 195 is its own cluster. They are not excluded and they are not adversarial; they are simply outside the reach of a rule that runs on co-spends.
Why is "at most 696 entities" a stronger claim than "about 696 entities", even though it sounds weaker?
Wrongly merged is 0 for a structural reason, so every error inflates and the figure can never be below the truth. That statement survives any amount of new data. "About" claims the truth is also not far below, which is exactly the direction with no evidence, and on month A it is wrong by 392.
Homework
Twenty minutes, on your own run and on the sentence you wrote in step 2.
Tabulate the five months without the truth
Addresses, ever spent as an input, merges and clusters for the five seeds. Add the never-spent column yourself. Then mark which single row you would have flagged as a problem if you had only seen this table.
Bring a guess about the operators
Four operators run 60 addresses each in this world. Write down, as a single number, how much smaller you think the clustered figure would get if every one of them co-spent carelessly rather than carefully. Session 5 measures it, and the answer surprises most people.