You wrote a number down. Here is the experiment that checks it.
Session 4 ended by asking for a single figure: how much smaller would the clustered count get if all four operators stopped being careful and co-spent like everybody else? Most people in the room write something large, because the operators are the deliberate part of the problem and deliberate things feel like they dominate. The number is measurable on Bench A in about fifteen lines, and tonight starts by building the measurement rather than by revealing it, because the way this experiment is assembled is the transferable part.
★ What you walk out with tonight. A counterfactual sweep built correctly, with the seed convention that makes it valid and the one mistake that silently invalidates it. The measured answer on month A: 666 at 50 percent discipline, 697 at 100 percent, a swing of 31 against a gap of 392 clusters. Three sources of over-count laid out side by side with an honest note that they are not a partition, including the 195 addresses on month A that never appear as a spend input at all. And the engineering conclusion that follows, which is that a smarter Sybil heuristic is aimed at the smaller part of the problem.
LiveDiscipline is a knob here and a guess in production4 min▶
Bench A has a field called sybilDiscipline. It is the probability that an operator sending a transaction avoids putting two of its own addresses into the same input list. At 1.0 an operator never co-spends and leaves no structural trace of common control. At 0.5 it co-spends about as often as an ordinary holder does.
Nothing in a warehouse exposes that field. You cannot read an operator's tradecraft off the rows, because the rows are the tradecraft: a careful operator's transactions look exactly like 60 unrelated people's transactions, which is the entire point of being careful. So the only place the question can be answered is a bench where the generator knows who owns what, and the answer transfers because it is a statement about the method rather than about this particular world.
Larkfield had a detection project scoped and costed before this sweep was run: heuristics on timing, on amount rounding, on funding patterns, six weeks of work for two engineers. The sweep took an afternoon and changed the scope, not because the detection ideas were bad but because it put a ceiling on what any of them could recover.
Self-studyThe words used tonight4 min▶
- Operator. A party deliberately running many addresses. Bench A has 4 of them, 60 addresses each, so 240 addresses of the roughly 1,000.
- Discipline. How reliably an operator avoids co-spending from two of its own addresses. Swept from 50 percent to 100 percent tonight.
- Counterfactual sweep. Re-running one pipeline many times with exactly one input changed and everything else pinned. The whole of tonight's method.
- Pinned seed. A generator seed held constant so that two runs describe the same world. Bench A uses world seed s and transaction seed s + 1.
- Swing. The distance between the two ends of a sweep. Here 31 clusters, and the quantity the sweep exists to produce.
- Ever spent. An address that has appeared at least once as a transaction input. 787 of 982 on month A.
- Structural error. Error present before any party does anything deliberate. Tonight's headline is that most of the error is this kind.
- Record linkage. Joining records about one party through fuzzy attributes and agreement weights, where extra features genuinely help. That craft is learn-entity-resolution-with-phoebe, and the evidence base there is different enough that its intuitions do not carry.
Pin four things, move one, and know which one you moved
15 min liveA counterfactual sweep is worth exactly as much as its pinning. Four stages run on every pass: the world, the transactions, the clustering, the score. Three of them must be byte-identical between passes or the difference you measure is not the difference you named. The one that varies is a single field in stage two, and the commonest way this experiment gets run wrong is to let the world move with it.
LiveWhy the pinning is the hard part4 min▶
The sweep is four lines of loop and the pinning is the rest of the file. Bench A takes two seeds because it has two generators, and the convention is fixed: the world is built from s and the transactions from s + 1. Keep both and a pass at 50 percent discipline and a pass at 100 percent discipline describe the same 982 addresses held by the same 304 parties, differing only in which input lists the operators built.
Let the world seed drift and the address count moves too, because the number of addresses each person holds is drawn from that generator. You would then be comparing 982 addresses at one end against some other number at the other, and attributing the whole difference to operator behaviour. The result is not noisy, which would at least be visible. It is confidently wrong in a direction that depends on the draw.
The general discipline here is not specific to chains and it is not specific to this bench: a counterfactual is a claim about one variable, and every unpinned input is a second variable you have quietly agreed to blame the first one for.
// Month A. The seed convention is part of the bench: world s, transactions s + 1.
var s = 11;
function pass(discipline) {
var world = W3.makeWorld({ people: 300, sybils: 4, sybilAddrs: 60, seed: s });
var txs = W3.makeTransactions(world, {
transactions: 4000,
coSpendRate: 0.25, // ordinary holders, pinned
sybilDiscipline: discipline, // the only field that moves
seed: s + 1
});
var cl = W3.clusterByCoSpend(world, txs);
return W3.scoreClustering(world, cl);
}
var careless = pass(0.50); // clustered 666
var perfect = pass(1.00); // clustered 697
// The number the afternoon was for:
var swing = perfect.clustered - careless.clustered; // 31
var gap = careless.trueEntities // 304 entities in this world
? pass(0.97).stillSplit : null; // 392 on the shipped setting
LiveSweep the thing you would actually change3 min▶
There is a second field in the same call, sybilAddrs, and sweeping it answers a different question: what happens if the operators run more addresses. That is a useful question and it is not this one, because it is not a lever your detection work can touch. An operator decides how many addresses to run and you do not get a vote.
Discipline is the field to sweep because it is the field a detection project is implicitly betting on. A heuristic that finds operator addresses is, in effect, a way of recovering links the operators declined to give you. Its best possible outcome is the careless end of the discipline sweep, where every link the operators could have leaked has already been leaked. So the careless end is not a scenario, it is a ceiling on the project.
That reframing is worth stating before the number appears, because it decides what the number means. The sweep is not a description of four hypothetical operators. It is a measurement of how much room a whole category of work has to move in.
31 clusters of room inside a gap of 392
12 min liveRun both ends and month A reports 666 with the operators careless and 697 with the operators perfect. Put that swing on the same scale as the error it was supposed to explain and the picture stops being about adversaries. The gap between what the method reports and what is true is 392 clusters. Operator behaviour moves 31 of them.
...
LiveRead the columns that do not move4 min▶
The clusters column moves and it is the one everybody reads. Two columns beside it do not move at all, and they carry more of the lesson. The address count is identical on every row, which is the pinning working: same world, same 982 addresses, same 304 parties. If that column ever varies down a sweep, stop and fix the harness before reading anything else on the screen.
The wrongly merged column is 0 on every row at every discipline setting, which session 4 derived and which the sweep now confirms under a condition session 4 never tested. Making the operators careless does not create a single false fusion. It only hands the method links it did not previously have, and a link that exists is a link between two addresses one party genuinely controls.
So the sweep does not trade one error for another. There is no operating point here, no curve, and nothing to balance, which is a real structural difference from the scored linkage taught in learn-entity-resolution-with-phoebe and worth saying out loud to anyone arriving from that background expecting a threshold.
LiveThree sources, and they are not a partition4 min▶
Where does an over-count of 392 come from? Three things contribute and it is tempting to draw them as three slices of a pie. Resist that, because two of the three are counted in different units and the third has no count at all on this bench, so a pie chart would be inventing the one thing the run does not contain.
The first is operator tradecraft, and the sweep measures it directly: 31 clusters, end to end. The second is addresses that never appear as a spend input at all, which session 4 counted at 195 of 982 on month A. Those are not badly clustered, they are unclustered: union takes inputs as arguments and never sees them. The third is ordinary holders whose addresses simply never happened to be spent together in the month, which leaves no artefact you could count directly.
What can be said with the numbers in hand is the comparison the sweep was built for. The adversarial component is 31. The total is 392. Whatever the exact anatomy of the remaining 361, it is not operators hiding, because the sweep already drove them from perfect concealment to none and collected 31 clusters for it.
LiveWhat this does to a detection roadmap3 min▶
The practical output of the afternoon is a ceiling you can put in a planning document. Any heuristic that works on the same rows, however clever, is trying to recover links the operators withheld. The sweep says that all of those links together are worth 31 clusters on month A. A project that delivered every one of them perfectly would move the reported figure from 696 to 666 and leave the reader with a number still more than twice the truth.
That is not an argument for doing nothing about operators, and it is important not to let it become one. Knowing which 240 addresses are operator addresses has uses that have nothing to do with the headline count: it changes who you pay, it changes a fraud picture, it changes what a cohort means. Those are real and they justify real work.
What the sweep kills is one specific claim, and it is the claim the project was funded on: that better Sybil detection will fix the active-address figure. It will not, and the number that says so is 31.
Self-studyWhere serious people disagree5 min▶
One camp says a figure like this should not be published externally at all. Their argument is specific to the adversarial setting rather than general squeamishness about error: publishing the number creates a target, the target is cheap to move, and the error is dominated by structure that no amount of care on your side will reduce. A bound does not rescue it, because a bound still gives the party who wants the figure higher a quantity to push on and a public statement of how you compute it. On this view the honest move is to report activity you can define without counting parties at all, and to keep the party count as an internal artefact with its limitations attached.
The other camp says non-publication buys nothing here, because the rows are public. Anyone who wants an active-address count can compute one from the same data, with a worse method, and will present it without a bound and without the 195 never-spent addresses named. Withholding yours does not remove the figure from circulation, it removes the careful version of it. On this view you publish, with the ceiling and the direction fused to it, precisely because you are the only party with an incentive to state the error honestly, and because a published method can be checked by the stranger who doubts it.
Both have watched the failure they are describing. Notice where they actually differ: not on the method, not on the 392, and not on whether the number is gameable, since both accept all three. They differ on whether a figure that will exist anyway is better with your name on it or without. That is a question about your position rather than about your pipeline, and it moves depending on whether anyone else is already publishing one.
Run the sweep, then break it on purpose
15 min hands onSix steps with the live box open. The middle two are the ones worth the time: a sweep that is wrong in the way most sweeps are wrong, run deliberately so you can recognise the output next time it happens by accident.
Reproduce both ends
World seed 11, transaction seed 12, discipline at 0.50 and then at 1.00. You should read 666 and 697. If you read anything else, the seed convention is the first thing to check and it is almost always the whole of the problem.
Assert the pinning before you trust a row
Write one assertion that the address count is identical across every pass of your sweep, and one that wrongly merged is 0 on every pass. Both should pass. Both are cheap. Neither is optional once somebody else can edit the harness.
Break it: move the world seed with the discipline
Run the sweep again with the world seed incremented on every pass. Record the swing you get. Then notice that nothing errored, nothing looked odd, and the number you would have quoted came out of a comparison between different worlds.
Say what your broken sweep would have justified
Take the swing from step 3 and write the sentence you would have put in the planning document. One line. This is the step people remember, because the sentence reads exactly as confidently as the correct one does.
Put the 31 against the 392 in your own words
One sentence, no jargon, aimed at whoever funds the detection work. It has to survive being repeated by them to somebody else, so it has to carry both numbers and the direction. Test it by saying it out loud.
Write the ceiling into the project, not the appendix
If detection work goes ahead, the best case belongs in the success criteria: the figure moves by at most this much. A project whose best outcome is written down in advance cannot quietly be judged later against an outcome it was never able to reach.
The habit to keep. Before funding work against an adversary, measure the adversary's room. Build the counterfactual where they behave as badly as they possibly could for their own purposes, and read what your metric does. If it barely moves, the adversary is not your largest error source and the work aimed at them is aimed at the smaller part of the problem, however satisfying that work is to scope. The measurement takes an afternoon and it is the cheapest thing in the plan.
Three questions
A colleague sweeps operator discipline and reports a swing of 140 clusters. You look at the harness and find the world seed incremented on each pass. What has been measured?
The world generator decides how many addresses each person holds, so moving its seed changes the population as well as the behaviour. The run completes, the table looks normal, and the number attributes a difference of two causes entirely to one of them.
Operator discipline goes from 100 percent to 50 percent and the wrongly merged column stays at 0. Why is that the expected result rather than a surprise?
Lowering discipline hands the method more true links, never a false one. The sweep therefore has no error trade-off in it at all: there is no operating point to choose and no curve to plot, which is a genuine structural difference from score-based linkage.
A detection project promises to identify operator addresses and thereby fix the active-address figure. What does the sweep say about the second half of that promise?
A heuristic on the same rows can at best recover the links the operators withheld, which is exactly what the careless end of the sweep already contains. Option C describes an outcome no rule reading co-spends can produce, because 240 addresses only collapse when the evidence to join them exists.
Homework
Twenty minutes. One sweep on something you own, and one number about your loader that session 6 will use.
Sweep an adversary in your own estate
Pick a metric somebody outside your team benefits from moving. Write down the most extreme version of their behaviour that is still rational for them, estimate what your metric reads under it, and compare that to your metric's known error. Bring both numbers, even if one is a rough order of magnitude.
Bring a guess about your warehouse
Session 6 measures what happens when an upstream withdraws history a loader has already written. Before then, write down one number: what percentage of the rows in a table you own would be wrong tomorrow if your upstream retracted its last three batches, and say how you would find out.