Master data, and the linkage you should not make

Session 3 rested on a named 1969 theory with a proof behind it. Tonight does not have that luxury, and pretending otherwise would be dishonest to you.

The honest framing for the rest of tonight: everything in Part 1 is what practitioners actually do and why it works, sourced to where it actually comes from.

It would be easy to teach master data first, since "merge the duplicates into one record" sounds like the whole job. It is not, because survivorship rules and architecture choices are meaningless until you already trust which records are duplicates - which is everything sessions 1 through 5 built. Master data is the last mile precisely because it is downstream of every earlier decision, and a golden record built on a bad match is worse than no golden record at all.

Once you know what a golden record should say, you still have to decide where it lives relative to the systems that fed it. Four styles cover essentially every real MDM programme.

The DAMA-DMBOK, second edition, is the closest thing this discipline has to a standard textbook, and it deliberately combines Reference Data and Master Data into a single knowledge area rather than treating them separately. It defines reference data as data that helps classify or categorize other data - country codes, status codes, product categories - which master data then draws on.

Splink is the tool this course would actually point a team at. It implements the same Fellegi-Sunter model session 3 taught, estimates the m and u probabilities with expectation-maximisation instead of guessing them, and runs on DuckDB, Spark or Postgres depending on scale. It is MIT licensed with no strings attached, actively maintained by a government team with a public track record, and it won a UK Civil Service Award for Innovation in 2025. Everything the bench on this page has been simulating in the browser, Splink does at real scale.

Zingg is AGPL-3.0. That is copyleft with real commercial implications, not a permissive licence like the other four tools here. AGPL's network clause means that if you run a modified version of Zingg as a service, even internally across a company boundary, you can be obligated to make your modifications available under the same licence. A team that adopts Zingg without a legal review, builds proprietary matching logic on top of it, and only reads the licence when a customer contract asks about it, is the exact six-months-later failure this course wants you to see coming instead.

dedupe's flag is different in kind: nothing about its MIT licence is a problem, but its most recently observed copyright notice only goes through 2022, and current maintenance could not be confirmed for this course. An unmaintained MIT tool is a smaller risk than a maintained AGPL one, but it is still a risk you should check yourself before depending on it.

Febrl is worth knowing for a different reason entirely: it is not maintained as software, but its synthetic benchmark datasets are still the reference point the field cites - Febrl1 (1,000 records, 500 pairs), Febrl3 (5,000 records), and Febrl4 (10,000 records, split into subsets 4a and 4b). The DBLP-ACM and Amazon-Google benchmarks, from the Magellan and deepmatcher research line, are used the same way; both are secondary references here, useful for finding real benchmark data rather than as tools to run.

Schnell, Bachteler and Reiher, BMC Medical Informatics and Decision Making 9:41, 2009, DOI 10.1186/1472-6947-9-41, is the primary, verified source for privacy-preserving record linkage with Bloom filters. Each field's q-grams get encoded into a bit array using cryptographic hashes, and similarity between two records is computed as Dice similarity on the bit vectors, without either party ever holding the other's plaintext. Reported match quality is comparable to matching on unencrypted identifiers, which is exactly why it is used in real cross-agency health linkage, where the whole point is that no single party should see identifiable data from the other side.

The honest second half: this technique has a real, actively studied attack surface. Frequency-based cryptanalysis exploits the fact that bigram and value frequencies leak through the bit patterns themselves - an attacker holding population name-frequency tables (common surnames appear often, rare ones appear rarely) can start re-identifying the frequent values from the pattern alone, even without breaking any hash.

The refusal this session is named for is not about Bloom filters, it is about a simpler fact: linking two previously separate datasets can turn data that was not reasonably identifiable on its own into identifiable personal data, and the act of linking is itself a new processing operation that needs its own lawful basis. A company that legitimately holds two datasets, each individually harmless, can create something legally sensitive purely by joining them - and Meridian's whole project is joining datasets.

What this means in practice for Meridian: the fact that a support system and two CRMs can be joined does not settle whether they should be, for a given purpose. A resolved customer view built to fix billing is a different processing decision from the same resolved view built to score creditworthiness or infer health status from purchase history, even though the underlying match is identical. The purpose the linkage serves is the thing that needs its own lawful basis, not just the technical act of matching.