Where we are
Session 3 gave you a match weight and two thresholds. Session 4 gave you a way to reach the candidates fast enough to score. Tonight is the last piece before session 6's golden records: how do you know any of it worked? The uncomfortable answer is that the most familiar metric in machine learning, accuracy, is close to useless here, and the field has quietly agreed to stop reporting it. You will watch the bench prove that on real numbers rather than take it on trust.
★ What you walk out with tonight. Pairwise precision, recall and F1, computed the way the whole field computes them. The specific reason accuracy misleads on imbalanced pair data, with Meridian's own numbers as proof. And the gap between judging pairs and judging the clusters those pairs become, including two ways transitive closure fails in opposite directions on the exact same dataset.
LiveThe scorecard everyone in the field actually uses4 min▶
Entity resolution treats scoring as a pair classification problem, and borrows its metrics from standard information retrieval practice rather than inventing new ones. For every pair the matcher decided on, there are four outcomes against the labels.
- True positive - the matcher said match, the label says match.
- False positive - the matcher said match, the label says no. A false merge.
- False negative - the matcher said no match, the label says yes. A missed duplicate.
- True negative - the matcher said no match, correctly. On Meridian there are 1,918 of these waiting to inflate any accuracy figure you compute.
Precision is true positives over everything the matcher called a match: of the pairs you merged, how many were right. Recall is true positives over everything that was actually a match: of the real duplicates, how many did you find. F1 is their harmonic mean, which only looks good when both are reasonably good, unlike a plain average.
A stakeholder asking "is the merge accurate" is asking the wrong question in the right words. The answer they need is two numbers, not one: how many false merges landed in the golden records, and how many duplicates are still sitting there unmerged.
Self-studyWhy this session, and not session 3, owns evaluation3 min▶
Session 3 built the scorer. This session judges what it produced, and deliberately keeps those two jobs apart. It is tempting to eyeball a handful of match weights and declare the system good, the way it is tempting to eyeball a few model predictions and call training done. The discipline is the same one that makes machine learning work at all: hold out the labels, compute the metrics, and let the numbers argue instead of your impression of the demo.
Why accuracy is the metric the field refuses to report
9 min liveThis is the spine of tonight. Meridian has 35 true pairs among 1,953 possible pairs, so 1.8 percent of pairs are matches. That ratio is not a quirk of a small teaching set, it is the shape of every real deduplication problem, and it gets worse as the data grows.
True pairs grow roughly linearly with the number of records, since each real person just adds their own duplicates. Total pairs grow with the square, since every record is compared against every other. Divide a linear thing by a quadratic thing and the ratio collapses: a model answering "no match" to every pair scores about 98.2 percent accuracy on Meridian, and would score above 99.99 percent on a million records, while finding not one duplicate. This is why the field reports precision and recall, and essentially never accuracy, for pair classification.
LiveThe anti-lever: a config that looks flawless and fails half the job5 min▶
Set the bench to compare on email only. It reports precision 100%, recall 51%, F1 0.679, accuracy 99.1%. It found 18 of the 35 true pairs, missed 17, and made zero false matches. Read the precision and accuracy tiles alone and this looks like the best configuration on the page. It is the worst one that ships zero false matches, because half of Meridian's actual duplicates never got merged.
The reason is structural, not a bug: plenty of true duplicates do not share an email at all, because one system holds a work address and another holds a personal one, or the field is simply blank. A single strict field can be perfectly precise and still miss most of the job.
Honesty check. "Pairwise precision, recall and F1" is standard information retrieval practice applied to pair classification, not a single citable paper - there is nothing to invent a citation for here, and nothing below pretends otherwise.
Self-studyThe other failure mode: exact name only4 min▶
Set the bench to match on exact name only and lower the threshold until it fires. It reports precision 80%, recall 23%, F1 0.356, with 2 false matches and accuracy 98.5%. The false matches are real people who happen to share a name: two different Maria Gonzalez, Robert Chen against Robert Chan, Samuel Adeyemi against Sam Adeyemo. Session 1 already showed you the Maria Gonzalez pair as a labelling trap; here it becomes a matcher's false positive, because a name is weak evidence on its own and this configuration trusted nothing else.
Notice the accuracy figure again: 98.5 percent, on a configuration that finds fewer than a quarter of the true duplicates and actively merges two pairs of strangers. Three configurations, three very different quality profiles, three accuracy scores clustered within two points of each other. That clustering is exactly why accuracy cannot discriminate between them.
A pairwise score is not a cluster score
6 min livePrecision, recall and F1 are computed on pairs. But the thing you actually deliver to the business is a set of golden records, one per real person, which means clusters. The gap between those two units of measurement is where a lot of quietly bad merges hide.
A single wrong pairwise merge can silently fuse two entirely separate true clusters into one, and a single missed pair can split a real person into two golden records. Neither cost scales with the count of pairwise errors, because clusters do not care how many wrong edges connected them, only whether they got connected at all.
Self-studyThe measure built specifically for this gap5 min▶
The dedicated cluster-level measure is B-cubed precision and recall, from the coreference-resolution literature: for every record, compare its predicted cluster against its true cluster, and average that agreement across all records rather than across all pairs. It penalises exactly the fusion and splitting failures pairwise F1 can miss.
Verification honesty. B-cubed traces to Bagga and Baldwin, 1998, and the survey "(Almost) All of Entity Resolution" (arXiv:2008.04443) covers it well - both citations are secondary here, not fetched and re-verified for this course. Treat the name and the idea as reliable, and confirm the exact formulation yourself before quoting it in a paper.
The bench on this page does not compute B-cubed; it reports precision, recall and F1 on pairs, plus a raw cluster count once you turn on transitive closure below, which is enough to see the two failure directions with your own eyes.
Turning pairs into clusters is its own decision, and it fails both ways
7 min liveOnce you have decided which pairs match, something still has to decide which records belong to the same entity. The obvious move is connected components: if A matches B and B matches C, put A, B and C in one cluster. That obvious move is naive transitive closure, and it is wrong in a specific, teachable way.
Connected components is naive transitive closure. Any chain of pairwise matches, however weak each link, merges into one cluster. It over-merges because pairwise match is not transitive: A resembling B and B resembling C does not make A resemble C, and a long chain of weak links fuses unrelated entities. This is the chaining problem, and Meridian shows both of its failure directions on the same 63 records.
| Configuration plus closure | Clusters found | Against true | What went wrong |
|---|---|---|---|
| Sensible config (upper 4) | 33 | 32 true | Under-merges: Sarah's third record never links to the other two |
| Permissive config (upper -3) | 28 | 32 true, 8 false pairs, precision 81% | Over-merges: chains unrelated people together |
LiveTurn on closure in the bench and watch both directions5 min▶
Run the sensible preset, then enable transitive closure. The cluster count lands at 33 against 32 true: it under-merges, because one of Sarah Whitfield's three records never links to the other two through any surviving edge, so she is split across two golden records. Now loosen the upper threshold to -3 and enable closure again: the count lands at 28 against 32 true, with 8 false pairs and precision down to 81 percent. It over-merges, chaining unrelated people through weak intermediate links.
The same dataset, the same clustering rule, two opposite failures depending only on where the threshold sat. That is the honest reason production ER systems rarely ship plain connected components without a second look at what it merged.
Self-studyWhat a formal clustering objective looks like5 min▶
Correlation clustering (Bansal, Blum and Chawla, Machine Learning 56, 2004, 89-113, DOI 10.1023/B:MACH.0000033116.57574.95) states the objective connected components only approximates: treat the pairwise decisions as a signed graph, a plus edge for match and a minus edge for non-match, and partition the records to minimise disagreements with those edges. It is the formal target ER clustering should aim at, and it is NP-hard in general, which is exactly why cheaper heuristics like connected components dominate practice despite the chaining problem.
Markov clustering (van Dongen, PhD thesis, Utrecht, 2000) is one such heuristic: simulate random walks on the similarity graph and let flow naturally strengthen within true clusters and evaporate between them. The algorithm itself is well established outside entity resolution; its use as an ER clustering step is the secondary, applied part of that citation.
| Clustering approach | Idea | Weakness on Meridian's data |
|---|---|---|
| Connected components | Any chain of matches merges | Chains James Whitmore senior and junior through a shared landline |
| Correlation clustering | Minimise disagreements on a signed graph | NP-hard, needs an approximation in practice |
| Markov clustering | Simulate flow, let it separate weak clusters | Primary for the algorithm, secondary as an ER application |
Score three configurations, then break clustering on purpose
★ 18 min · everyone buildsThe bench below is the real matcher over Meridian's 63 labelled records. Toggle fields, pick a name comparison, choose a blocker, move both thresholds, and it recomputes precision, recall, F1, accuracy, the clerical band size, the blocking metrics and the cluster count live against the labels.
Click "A sensible configuration". Record precision, recall, F1 and accuracy. Note that accuracy and F1 disagree with each other about how good this is, and decide which one you would actually trust.
Click "Match on email only". Record the same four numbers. Precision and accuracy both look excellent. Recall does not. Write down, in one sentence, why a stakeholder shown only the top two numbers would approve shipping this.
Click "Exact name only". This time precision itself drops, to 80 percent, alongside a much lower recall. Find the two false matches the labels flag and name the real-world reason each pair of strangers shares a name.
Back on the sensible configuration, find the two true pairs it misses. These are the two false negatives behind its 94 percent recall. Note which fields on those specific records made them harder than the rest.
Turn on transitive closure with the sensible configuration still active. Watch the cluster count move to 33 against 32 true people. That is an under-merge, not an over-merge, even though closure is usually blamed for the opposite.
Loosen the upper threshold toward -3 with closure still on. Watch the cluster count move the other way, to 28 against 32 true, with 8 false pairs appearing. Same dataset, same clustering rule, opposite failure. Then decide: if you could hand a stakeholder exactly one metric from tonight, which one, and what would you refuse to also show them without it?
Notice that "the cluster count moved toward 32" is not automatically good news. 33 and 28 are both wrong, in opposite directions, and a dashboard that only reports "number of golden records: 30" would look reasonable while hiding either failure completely.
Before session 6
25 min- Write a one-paragraph scorecard for the sensible configuration, aimed at a stakeholder who will ask "is it accurate". Use precision, recall and F1. Do not use the word accuracy without immediately explaining why it is misleading here.
- For the email-only configuration, write the sentence a vendor might use to sell it ("100 percent precision, 99.1 percent accuracy") and then the sentence that corrects it.
- Pick one of the two closure failures, under-merge or over-merge, and describe in plain terms what a customer would actually experience because of it - a duplicate welcome email, or a support ticket routed to the wrong household.
- In one sentence, state whether you would ship connected components clustering to production as-is, or add a second check, and what that check would look for.
Three questions
A model answers "not a match" to every pair in Meridian. What does it score, and why does the field avoid this metric for pair classification?
It gets every one of the 1,918 true non-pairs right and finds none of the 35 true pairs, landing at roughly 98.2 percent accuracy while doing zero useful work. The ratio only worsens as data grows, which is why precision and recall on matched pairs are reported instead.
Matching on email only scores 100 percent precision and 99.1 percent accuracy on Meridian. What is wrong with presenting only those two numbers?
Email-only is the anti-lever of the session: it makes zero false matches and looks flawless on precision and accuracy while finding only 18 of 35 true pairs, because plenty of genuine duplicates do not share an email at all. Recall is the number that exposes this, and it is exactly the number a favourable report would omit.
Why can turning on transitive closure make the cluster count move in either direction depending on the threshold, on the very same 63 records?
The sensible configuration plus closure gives 33 clusters against 32 true, under-merging because one of Sarah Whitfield's records never links. A permissive threshold plus closure gives 28 clusters against 32 true with 8 false pairs, over-merging by chaining unrelated people. Same clustering rule, opposite failure, because A matching B and B matching C never guaranteed A matches C.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 5 cheat sheet · pin this
The golden record is the thing all five of these sessions have been building toward, and it turns out to have no academic canon at all - just vendor vocabulary that happens to be genuinely useful. Session 6 covers survivorship rules, the four ways to architect a hub, the tools that actually exist in 2026, and the one kind of linkage you should refuse to make even when you technically can.