Information economics of models

The setup needs only two facts. Quality varies between sellers, and the buyer cannot observe it before paying. With those two, a buyer's best offer is the price of the average good, because that is the expected value of what she is about to receive. That offer is a bargain for the seller of a bad good and an insult to the seller of a good one, so the good one withdraws. Now the average is lower, the buyer's best offer is lower, and the next-best seller withdraws. Akerlof's contribution was to show that this does not settle at some middling equilibrium; under simple assumptions it unravels all the way, and the market for good goods disappears even though there are buyers who would pay for them and sellers who would sell.

Datasets fit the setup exactly. A labelled dataset's quality is its label accuracy, its coverage of the cases you care about, its freshness and its provenance, and none of those is on the listing. You find out after you have trained on it. A vendor with clean labels and a vendor with labels produced by an underpaid contractor at speed both say "high-quality, human-annotated", and both charge what the market will bear for that sentence. The clean vendor's costs are higher. At a price set by the average, she is the one who cannot make it work.

Models fit it too, with one twist. A model's headline score is visible, so it looks as if quality is observable. But the score is on a benchmark the seller chose, possibly trained toward, and the buyer's data is not that benchmark. What the buyer needs, performance on her own cases, is hidden until she has integrated the thing. The visible score is not quality; it is a signal of quality, sent by the seller, which is the subject of Part 2.

Akerlof's paper lists the counteracting institutions: guarantees, brand names, chains, licensing. Each is a way of making quality partly observable or of making the seller bear the cost of a lemon. The data market has grown the same set under different names. A guarantee is a service-level agreement on label accuracy with a refund clause. A brand is a vendor whose past datasets you or people you trust have used. Licensing is an audit by a third party who is paid by neither side. Each one costs something, and that cost is the point: a signal that is free to send carries no information, because the lemon seller sends it too.

The institution that matters most for data and is hardest to build is the sample. A buyer who can inspect a random slice of the dataset before paying, labelled, with the sampling done by the buyer and not the seller, has converted a hidden quality into a measured one, and the spiral stops for her. The reason it is hard is that the sample has to be drawn by the buyer from the whole, not selected by the seller from the best part, which requires the seller to hand over the whole before the sale, which is the thing being sold. The workable compromises, escrow, a third party who draws the sample, are the data market's version of the used-car inspection.

A published benchmark score is a signal the seller sends. When the benchmark is new, scoring well is expensive: it takes a model that is actually good at the thing. That cost is what makes the score informative, because a seller of a lemon cannot afford to send it. The cost then falls, for three reasons that all happen at once. The test set leaks into training data, on purpose or by scraping. Tuning choices get made by watching the score, which fits the model to the benchmark's quirks. And the benchmark's own gaps become known, so effort goes where the score is and not where the buyer's cases are. At the end of that process the score is cheap to send and everyone sends it.

A private eval is a screen the buyer controls. A held-out set the seller has never seen, drawn from the buyer's own distribution, cannot be tuned toward, and it measures the thing the buyer will actually get. It costs the buyer something to build, which is why so few buyers do, and that cost is exactly what makes it work: a signal that was free would be sent by everyone. The mechanics of building one, what to hold out, how to score it, how it drifts, are the subject of the AI evals course at learn-ai-evals-with-phoebe, and tonight does not teach them; it teaches why the eval is the buyer's side of the contract.

An audit by a third party is a signal with a borrowed cost. The seller pays for it, but the auditor's reputation is what is at stake, so the seller cannot make the audit say what she likes. It is informative in proportion to the auditor's independence and to how much of the model the auditor actually got to see, both of which the buyer should ask about before trusting the stamp.

The sentence everyone quotes is "when a measure becomes a target, it ceases to be a good measure". It is almost always attributed to Charles Goodhart, and the law does carry his name, but that wording is not his. It is Marilyn Strathern's, in "Improving ratings: audit in the British University system", European Review 5(3) 305-321 (1997), where she was summarising the idea in the context of academic audit. Goodhart's own 1975 formulation was about monetary policy and is phrased quite differently. The distinction matters for the same reason every citation on these pages matters: attributing a sentence to someone who did not write it is a small lemon of its own, and a reader who catches it will discount the rest.

The mechanism the sentence names is the one in the figure. A measure is useful because it correlates with the thing you care about. Make it a target, pay on it, and the people being paid find the cheapest way to move it, which is usually not the way that moves the thing you care about. The correlation that made the measure useful was never a law of nature; it held only while nobody was pushing on it. The public benchmark curve keeps rising because pushing on it is what everyone is paid to do. The held-out curve flattens because nobody can push on it, which is the only reason it is still a measure.

The two words get used interchangeably and they are not. In signalling, the informed party moves first: the vendor publishes the score, writes the model card, commissions the audit. The buyer reads the signal and infers. The signal works if sending it is cheaper for a good seller than for a bad one, so that only good sellers find it worth sending. In screening, the uninformed party moves first: the buyer designs a test, a trial, a menu of contracts, and the seller's choice among them reveals what the seller knows. A vendor offered "flat fee" or "fee contingent on held-out accuracy" and choosing the flat fee has told the buyer something without a word being said.

The practical difference is who bears the cost and who controls the instrument. A signal is on the seller's terms: her benchmark, her card, her auditor. A screen is on the buyer's: her held-out set, her trial period, her contract. Data and model buyers who rely only on signals are reading what the seller chose to show. The buyers who do well add one screen, and the cheapest screen that works is a held-out set of their own cases that the seller never sees.

Put the two halves together and a private eval is not a technical artefact; it is a contract term. The buyer says: performance on this set, which you will not see, at this threshold, is what I am paying for. The seller who accepts has revealed that she believes her model generalises, which is exactly what a headline score could not tell you. The seller who negotiates the set down to something she has seen, or the threshold down to something the benchmark already guarantees, has revealed something else.

Inside a company the same contract exists between an ML team and the leader who funds it, and it has the same failure mode: if the team can see the eval, it becomes a benchmark, and the curve in the figure starts. The held-out set stays a measure only while it is held out from the people being judged on it, which in practice means it is drawn and scored by someone who is not on the team, refreshed on a schedule, and never used for tuning. Every one of those conditions costs something, and each cost is the reason the measure still works.

The principal has a goal and cannot observe effort or, in time, the outcome. The agent has effort to allocate and a reward that depends on something observable. A contract that pays on a proxy metric solves the observability problem and creates the Goodhart problem in the same stroke. The offline score in the figure is constructed; the pattern is not. A fraud model rewarded on offline AUC will be tuned for offline AUC, on the historical distribution, with the thresholds that flatter it, and the business outcome, bad approvals next quarter on a distribution that has shifted, is a different quantity that nobody was paid on.

The honest framing is that no one is behaving badly. The team is optimising the number the contract named. The leader chose the number because it was available. Both are doing what the structure asks, and the structure is what has to change. The economics gives three levers and none of them is "ask people to care about the real goal". Pay on something closer to the outcome, at the cost of a longer lag. Hold out a measure the agent cannot see, at the cost of building and refreshing it. Or accept the proxy and pay less on it, so the incentive to game is smaller than the incentive to do the job.

Where this meets Part 2: the buyer's held-out eval and the leader's held-out metric are the same instrument. In both cases the measure stays a measure only while the party being judged on it cannot push on it, and the moment it becomes visible to them the curve in the Part 2 figure starts again.

Two words from the same family are worth keeping apart. Adverse selection is hidden information before the contract: the dataset vendor knows her labels are poor and you do not, so the wrong vendors are the ones who accept your price. Moral hazard is hidden action after the contract: the team knows how much care went into the validation and you do not, so the contract's terms change what care gets taken. Part 1 was adverse selection. Part 3 is moral hazard. The lemons spiral is about who shows up; the metric problem is about what they do once they have.

Both have the same root, one side knows something the other cannot check, and the same family of cures, make it checkable or make the informed side bear the cost of being wrong. But they are cured at different moments. Adverse selection is addressed before the deal, with screens and samples. Moral hazard is addressed in the contract, with what is paid on and what is held out. A team that has a strong procurement process and a weak metric has solved the first and not the second, and will hire good vendors to build models that game the number.