How this course works
Six sessions, 45 minutes each, one artifact: a working matcher over Meridian, a company merging two CRMs and a support system into one customer master. Tonight frames the problem and the labels. Session 2 compares strings, session 3 scores pairs properly, session 4 makes it scale, session 5 judges it honestly, and session 6 decides what a golden record is and which linkages you should refuse to make.
★ What you walk out with tonight. The five stages of a resolution pipeline and which of them can lose a pair permanently. The arithmetic that makes brute force impossible sooner than people expect. And a labelled set you can actually score against, which is the difference between tuning a matcher and guessing at one.
LiveThe join that is not there4 min▶
Every join you have ever written assumed a key that both sides agreed on. Entity resolution is the work that starts when there is no such column, and the only evidence that two rows are the same person is that they look similar. The moment you accept that, three things follow that never apply to a normal join.
- The answer is probabilistic. Two rows are not equal or unequal, they are more or less likely to be the same person. Some pairs will be genuinely undecidable from the data you hold, and no amount of cleverness changes that.
- The work is quadratic. A join walks an index. Comparing every row against every other row grows with the square of the table.
- It has to be measured against labels. Without a set of pairs you know the answer for, "the matcher got better" is an opinion.
Meridian's finance team reports 4,100 active customers. Sales reports 3,600. Neither is wrong about its own system, and both numbers go in the same board pack. Nobody can say which is closer to the number of actual companies, because nothing in either system knows that Northwind Ltd and Northwind Limited are one account.
Self-studyThe vocabulary for the whole arc4 min▶
- Record - one row in one system. Entity - one real person or company, which may appear as several records.
- Pair - two records being considered. The unit of almost all the arithmetic in this course.
- Blocking - cheaply throwing away pairs that are obviously not worth comparing. Session 4.
- Match weight - a score for how much the evidence favours a pair being the same entity. Session 3.
- Clerical review - the band of pairs the system refuses to decide, which go to a human. Session 3, and the whole point of the course.
- Golden record - the single surviving version of an entity, assembled from the records that matched. Session 6.
One term deliberately avoided: "deduplication". It implies the duplicates are obvious and the job is deleting them. Neither is usually true, and the framing hides the decision that matters.
Five stages, and only one of them is reversible
7 min liveEvery resolution system, from a hand-written script to Splink, has the same five stages. Knowing which stage can throw away a true pair for good is what stops you tuning the wrong one for a week.
LiveWhere the field started, and why it was medical4 min▶
The first published probabilistic approach is Newcombe, Kennedy, Axford and James, "Automatic Linkage of Vital Records", Science, October 1959. The problem was linking birth, marriage and death records for Canadian health statistics, where there was no shared identifier and the stakes were epidemiological rather than commercial.
Their move was the one this whole course rests on: rather than asking whether two records agree, compute how much each field's agreement should shift your belief, and combine those shifts. Ten years later Fellegi and Sunter turned that into a formal decision theory, which is session 3.
It is worth knowing the origin is 1959 and not 2015. Entity resolution gets sold as a machine-learning problem, and the learning part is genuinely useful, but the core model predates it by decades and is still what the good tools implement.
Self-studyWhy "just clean the data first" does not work4 min▶
The instinct is to normalise everything until the rows match exactly, then join. It helps, and session 2 does a lot of it, but it cannot finish the job for a structural reason: normalisation is lossy and you do not know in advance which loss matters.
Fold "Katherine" and "Catherine" together and you correctly join one pair in Meridian and incorrectly risk joining two different people elsewhere. Strip company suffixes and Northwind Ltd matches Northwind Limited, which is right, and also matches a different Northwind that is a separate legal entity, which is wrong. Every normalisation is a bet about which differences are noise.
The probabilistic framing keeps the bet visible and priced. Instead of destroying the difference, you say how much a difference on this field should count, and that number is estimable from data.
The pairs grow as the square, and they grow fast
6 min liveMeridian is small enough to compare exhaustively, which makes it a good teaching set and a misleading one. The number of pairs is n times n minus one over two, and that expression stops being comfortable at a size most companies passed years ago.
LiveThe other consequence: almost every pair is a non-match5 min▶
In Meridian there are 35 true pairs among 1,953 possible ones, so 1.8 percent of pairs are matches. That ratio does not stay still as the data grows, it collapses: true pairs grow roughly in line with the number of records, while total pairs grow with the square.
This has a consequence people trip over constantly. A model that answers "not a match" to every single pair is right about 98.2 percent of the time on Meridian, and would be right about 99.99 percent of the time on a million records. Accuracy is not a usable metric here, and session 5 shows the bench reporting it next to F1 so the gap is visible rather than described.
A vendor demo that reports "99.4 percent accurate" on a deduplication task has told you almost nothing, and may have told you it does not work. Ask for precision and recall on matched pairs, and ask how many true pairs the labelled set contained.
You cannot tune what you cannot score
6 min liveMeridian comes with an answer key: every record carries the id of the real person it describes. That is what turns every later choice into a measurement. Getting such a key for your own data is manual, boring, and the highest-value work in the project.
63 records, 32 real people. Records sharing a true id are the same person.
LiveWhat the dirt actually is5 min▶
Read the table above and you can see every failure mode this course has to handle. None of it is exotic; all of it is what happens when humans type into three systems over five years.
| Kind of difference | In Meridian | Handled in |
|---|---|---|
| Nickname | Robert and Bob Chen, Michael and Mike Okonkwo | Session 2, normalisation |
| Spelling of the same sound | Priya Nair and Priya Nayar, Smith and Smyth | Session 2, phonetics |
| Transliteration | Thomas Muller and Thomas Mueller | Session 2 |
| Married or changed surname | Sarah Whitfield and Sarah Whitfield-Barnes | Session 3, weights |
| Token order | Wei Zhang and Zhang Wei | Session 2, token sets |
| Transposed digits | Chidi Nwosu's phone, 780 against 708 | Session 2, edit distance |
| Abbreviated street | Alder Street and Alder St, Court and Ct | Session 2, normalisation |
| Work against personal address | Elena Petrova's two emails | Session 3, weights |
| Missing value | Aisha Bello's second record has no email | Session 3, carries no evidence |
| Different people who look alike | Two unrelated Maria Gonzalez, Robert Chen and Robert Chan | Session 5, false matches |
| Genuinely undecidable | Two James Whitmore sharing a landline | Session 3, the clerical band |
Self-studyHow to get labels for your own data5 min▶
You will not label every pair, and you do not need to. What works:
- Sample from the middle, not the ends. Pairs that obviously match and pairs that obviously do not teach you nothing. Label the ones a rough first pass scored near its threshold, because that is where every future decision lives.
- Label pairs, not records. The question "are these two the same" is answerable. "Which entity is this" is not, until you already have the answer.
- Two labellers on a subset. Where two careful humans disagree, no matcher will do better, and that disagreement rate is a floor on your achievable accuracy. It is also the honest size of your clerical band.
- Record why. A label with a one-line reason survives the person who made it leaving. A bare yes or no does not.
- Keep it frozen. A labelled set you quietly fix whenever the matcher disagrees is not a measurement any more, it is a mirror.
The dedupe library builds its training set this way deliberately, asking a human to label the pairs it is least sure about rather than a random sample. That is active learning, and it is the right instinct even when done by hand in a spreadsheet.
Label five pairs by hand, then check yourself
★ 16 min · everyone buildsBefore any algorithm, do the job manually. It takes four minutes and it changes what you expect from the rest of the course.
In the table above, find r01 and r02. Decide: same person or not? Write down which fields convinced you, and in what order you looked at them.
Now r03 and r04, Katherine and Catherine Doyle. Same phone, same address, different first letter of the first name, different email domain. Decide, and note how confident you are as a percentage.
Now r05 and r59. Both are called Maria Gonzalez. Look at the emails, phones and addresses before deciding. They are different people, and the name is the field that misled you.
Now r13 and r63, two records for Sarah Whitfield. Only the mobile agrees; the email, address and company all differ because she changed jobs and moved. Decide, and note your confidence.
Finally r61 and r62, two records for James Whitmore. Same name, same landline, different companies, different addresses, different emails. Decide. If you feel you cannot, that is the correct answer, and session 3 gives that feeling a formal home.
Compare your five answers against the true ids in the table. Then count: which fields did you actually use, and did you weight them equally? Almost nobody does. You leaned on the phone and distrusted the name, and session 3 turns that instinct into estimated numbers.
Keep the note about your confidence on the Doyle and Whitmore pairs. In session 3 you will find that the model puts one of them well above its match threshold and the other in the band it refuses to decide, and it is worth knowing whether your own judgement agreed before you saw the arithmetic.
Before session 2
25 min- Take two systems in your own company that hold the same customers or people. Pull 40 records from each and put them in one sheet.
- Count the possible pairs with n times n minus one over two. Then work out what that number becomes at ten times the volume.
- Label 20 pairs by hand, chosen so that about half are hard. Record a one-line reason for each. This is your answer key and everything later is scored against it.
- Write down which single field you trusted most while labelling, and which field lied to you at least once. Session 3 estimates exactly those two things as numbers.
Three questions
Which pipeline stage can lose a true pair in a way no later tuning can recover?
Blocking is the irreversible stage. Scoring and the decision are recoverable by moving a threshold, and clustering can be rerun. A pair blocking drops is never scored, so no amount of threshold tuning brings it back. That is why session 4 measures blocking by pair completeness first.
Meridian has 35 true pairs among 1,953 possible pairs. What does a model that answers "not a match" to every pair score?
It gets every one of the 1,918 non-pairs right, so accuracy is about 98.2 percent while it finds nothing at all. The ratio only gets worse as data grows, which is why the field reports precision and recall on matched pairs instead.
Records r05 and r59 are both "Maria Gonzalez" and are different people. What does that establish?
Two unrelated people share a common name, which is exactly why an agreement on name shifts belief far less than an agreement on a phone number. Session 3 estimates that difference as the ratio of two probabilities rather than asserting it, and dropping the field would throw away real evidence.
What this session teaches, and where it came from
Every fact on this page, and its verification tier, is recorded in the course's source map.
Session 1 cheat sheet · pin this
Before any pair can be scored, "how similar are these two strings" needs an actual number. Session 2 builds seven of them, from edit distance to the 1918 Soundex patents, and finds the case where each one is confidently wrong - including the one where Jaro-Winkler's famous prefix boost cannot help at all.