ML on public adversarial data

Plenty of datasets are noisy, imbalanced or drifting, and the craft for all three is well developed. None of that craft is what is being tested tonight. Noise is a property of measurement. What Larkfield has is a property of authorship: some of these rows exist because a party decided they should exist, and that party knows what Larkfield can compute from them, because the columns are public.

That is a narrow difference and it has wide consequences. It means the training distribution is not a sample from a population, it is a population that was partly assembled. It means an evaluation set drawn the same way inherits the same composition. And it means a feature's usefulness is not a stable fact about the world, because the thing the feature reads is produced by somebody who can also read the feature.

• Unit of analysis. What one row of the training table represents. Tonight it is an address, a clustered entity, or a real party, and the three give different answers to the same question.
• Grain. The same idea said in a warehouse accent. Worth keeping both words, because the modelling side and the pipeline side rarely notice they are talking about one decision.
• Population composition. Who is in the table and in what proportion. On ordinary data this is an accident of collection. Here part of it is a choice somebody else made.
• One-sided error. An error that can only run in one direction. Bench A's clustering never fuses two parties, on any seed, so the count it produces can be too high and can never be too low.
• Observable features. Features computed from columns that anyone can read. Not a security property, a modelling one: the inputs and the blind spots are both public.
• Finality lag. How far behind the tip a loader stays before writing a block's rows. Session 7 priced it. Tonight it is a property of the training set rather than of the report.
• Operator. A party running many addresses deliberately. Bench A has 4 of them holding 60 addresses each, and the generator knows which is which so the truth can be checked.

Feature engineering on internal telemetry is a private act. You decide what to log, you decide what to derive, and the people who produce the events cannot see either decision. Here the derivation is the only part that is private, and even that is not much of a secret, because the co-spend signal is the standard one and anyone working this data reaches for it.

The sharper point is the blind spot rather than the feature. On seed 11, 787 of 982 addresses ever appear as a spend input at all. The remaining 195 are seen only as recipients, so the one feature available says nothing whatsoever about them. That is an ordinary coverage gap with one unusual property: anybody can compute exactly which 195 they are, from the same rows, with no access to Larkfield's warehouse.

A coverage gap you can enumerate from the outside is a different object from a coverage gap that lives in your logs. It is not a reason to stop using the feature. It is a reason to write down what the feature cannot see, in the model card, in the same place you write down what it can, because somebody else already has that list.

A table of one row per address looks like a sample. It is not. Across all five seeds of Bench A, exactly 240 addresses out of roughly a thousand belong to 4 operators running 60 each on purpose. That is about 24 percent of the training rows, and the 24 was not a fact about the world, it was a decision made by four parties who could have chosen 20 or 600.

This is why the word manufactured is worth keeping instead of softening it to imbalanced. Class imbalance is something you correct for, because the imbalance in the sample is an artefact of collection and the population underneath is fixed. Here the population underneath is not fixed. Reweighting to the observed composition reweights to a number somebody else set.

And the standard remedy does not remove it. Clustering by co-spend brings the count from 982 down to 696 on seed 11, which is an improvement, and the improvement is a ceiling rather than a correction. The truth is 304 and the clustered figure is still 2.3 times it. Session 4 measured that; tonight the consequence is that the training table is still mostly made of rows that do not correspond one to one with parties.

Session 7 priced the finality lag as a reporting decision. It is also a training decision, and it is a worse one, because a report is republished and a frozen training set is not. Bench B's naive loader ends the run with 511 rows, 5.75 percent of the warehouse, describing blocks that were replaced. A training set cut from that warehouse has those rows in it.

What makes this different from ordinary label noise is that the rows are not wrong in a random direction. They are all the tail of the table, the most recent blocks, which is precisely the region a time-based split puts in the validation fold. So the defect concentrates in the part of the data that is being used to decide whether the model works.

The fix is not a modelling fix and it costs nothing clever: apply a lag before you cut the snapshot, record the lag you used alongside the snapshot, and treat a training set with no recorded lag as a training set of unknown composition. That is a line in a manifest, and almost nobody writes it.

One camp says that a model should be trained at the grain the data actually has, which is the address, and that any attempt to model entities is importing an inference error into the features. Their case is concrete: the clustering is a heuristic with a known one-sided error, so training on clustered rows means training on a population whose composition depends on how well the heuristic happened to work in that month. Address grain is at least a stable definition. Whatever it fails to capture, it fails to capture the same way every time, and a stable wrong unit is easier to reason about than a unit that moves.

The other camp says that a model whose rows are not parties cannot answer a question about parties, and most of the questions worth asking are about parties. Training at address grain produces a model that is well calibrated on a population nobody cares about, and every downstream user will quietly read its output as being about people, because the output will be labelled with something that sounds like a person. On this view the inference error in the clustering is real and it is the smaller error, and the right move is to model at entity grain and carry the bound through to the output.

Both camps are staffed by people who have shipped models on identity-ambiguous data. Notice that Bench A does not settle the argument and cannot: it shows the clustering is a ceiling with zero wrong merges, which supports the second camp, and it also shows the ceiling is 2.3 times the truth and moves between 676 and 707 across months, which supports the first. What the bench does settle is that nobody may hold either position without stating which grain they are on, because the two produce different numbers on the same rows.

An evaluation set drawn from the same rows has the same composition as the training set, which is the whole point of drawing it that way and is usually correct. It stops being correct when the composition is not a property of the world. A score averaged over address rows on seed 11 is a weighted average in which the four operators carry 24.4 percent of the weight, and on clustered rows they carry 34.3 percent. Among real parties they are 1.3 percent.

So a model that handles operator traffic well and ordinary traffic poorly gets a good score, and the score is not lying about anything it claims to measure. It is an honest average over the rows it was given. The rows were the decision.

The second half is the direction. Bench A reports zero wrongly merged clusters on all five seeds, which means the clustering error only ever fails to join addresses that belong together. A count taken from these rows can come out too high and cannot come out too low. Any metric built on top of that population inherits the same one-sidedness, so an evaluation that looks well calibrated is calibrated against a ceiling.

The last consequence is the plainest and the one with no technical remedy. Larkfield's product issues a reward for activity. If a model's output decides who receives a reward, then the model's output is worth something, and something worth money on a public dataset will be optimised against by whoever it pays. Bench A already contains the shape of that: four parties running 240 addresses, and the generator does not need to know why they did it for the count to be wrong.

What is worth stating precisely is the boundary of what the bench shows. It shows that a population can be substantially composed by a small number of parties, that the standard remedy leaves most of that composition in place, and that the composition is invisible in any metric computed at address grain. It does not show any particular method anybody used, and nothing here is a description of how to do it.

The operational consequence is a governance one rather than a modelling one. A model whose output moves money needs the question "who benefits if this score is wrong in this specific direction" answered in writing before it ships, and the answer belongs next to the evaluation rather than in a risk register somebody reads annually. If nobody can name the beneficiary, that is not reassurance, it is the same gap that made the address count look like a user count.

Most model documentation templates have a field for the training data and it is usually filled in with a table name and a date range. On this data, three fields decide whether the document is worth anything, and none of them is standard.

The first is the unit of analysis, stated as a noun and not as a table name. "One row per address" and "one row per clustered entity" are different models with different meanings and the same schema, and the widget above shows they have different populations. A reader who cannot tell which one they are holding will read the output as being about people, because that is what the surrounding sentences will be about.

The second is the load discipline of the snapshot: the finality lag applied before the rows were frozen, as a number, with the reorganisation depth it was chosen against. A snapshot with no recorded lag is a snapshot of unknown composition, and Bench B says how unknown: 5.75 percent of a naive load at depth 3, concentrated in the most recent blocks.

The third is the direction of the known error. Not its magnitude, which moves between months, but its sign, which does not. Recording that the entity count is a ceiling with zero observed wrong merges tells a reader something they can act on immediately, which is that any downstream figure derived from it may be revised downwards and will never be revised upwards. That single sentence prevents more misuse than a page of metrics.