You Cannot Deduplicate Before You Normalise: Identity Is a Decision, Not a Property

Mostafa Dhouib·7 min read

You Cannot Deduplicate Before You Normalise: Identity Is a Decision, Not a Property

The short answer. When several sources supply overlapping inventory, the same underlying thing arrives more than once under different field names, different identifiers and different conventions. Removing those duplicates is impossible until every source has been translated into one internal shape, because two records in different shapes are not comparable at all. That makes deduplication structurally downstream of normalisation. The second half is harder and gets skipped: once the records are comparable, you still have to decide which fields make two of them the same thing. Nothing in the data answers that. On an integration layer, normalising first took duplicates from 12.2 percent of returned results to 1.2 percent, and the remaining 1.2 percent is where the identity decision lives.

Duplicates in a multi-source result set get treated as a cleanup problem. Something at the end filters them out, usually written after the first complaint.

They are not a cleanup problem. They are a consequence of integrating sources without a common representation, and the place to fix them is upstream of where they are noticed.

Why comparison requires a shape

Two providers describe the same thing and nothing about their descriptions matches.

Field names differ. Identifiers are provider-local and mean nothing to each other. Units and formats differ, so one supplies a duration in minutes and another a start and end timestamp. Precision differs. Optionality differs, so one always supplies a field the other omits when it is absent.

Given two records in those two shapes, a comparison is not merely hard, it is undefined. There is no field to compare because there is no shared field.

So the only place deduplication can happen is after every source has been translated into one internal representation. That is an adapter per source presenting a canonical shape, which people usually build for a different reason, and this is the second thing it buys.

Then identity is still a choice

Once the records are comparable, the real question appears and it has no technical answer.

Which fields make two records the same thing?

Two itineraries with the same flight numbers and times but different fare rules: one result or two? Two product listings with the same identifier but different availability: one or two? Two records with the same values and different providers, where one is bookable and the other is not?

Nothing in the data decides this. It is a product decision wearing an engineering costume, and it has consequences in both directions.

Too narrow an identity leaves duplicates. The user sees what looks like the same thing twice, which reads as a defect even when both records are legitimately different.

Too broad an identity merges things that differ in a way the user cares about. Two itineraries merged on schedule, where one had a changeable fare, means the user is shown one option and loses the one they would have chosen. This error is worse, because it is invisible: nothing in the output indicates a merge removed something.

Which record survives

Having decided two records are the same, you keep one. That choice is also a decision, and the obvious rules are each wrong somewhere.

Keeping the cheapest is correct when price is the only dimension and wrong when the cheaper record is the one that is not actually bookable. Keeping the first to arrive makes your output depend on provider latency, which varies. Keeping the most complete biases toward whichever provider is most verbose rather than most accurate.

The version that holds is to merge rather than choose: build a canonical record carrying the best-supported value for each field, with provenance per field. That costs more and it is the only approach where a survivor is not quietly worse than a record you discarded.

Keep the provenance. When a user or a support ticket asks why a result shows a particular value, the answer should be a trace rather than a reconstruction.

Measure both directions

The duplicate rate is easy to measure and it is half the picture.

The merge-error rate, meaning how often deduplication removed something materially different, is hard to measure and matters more. One approach that works: periodically keep both records instead of merging, compare them on the fields your identity rule ignores, and count how often they differed in ways a user would have cared about.

Without that second number, tightening deduplication always looks like progress. The duplicate rate falls monotonically as identity broadens, right up to the point where you are merging genuinely different things, and nothing in the first number ever objects.

Where this generalises

Anywhere records about the same real-world thing arrive from several places.

Customer records across acquired systems, where identity is the classic hard problem. Product catalogues from multiple suppliers. Threat or incident feeds. Scientific or clinical records across institutions. Search results across providers.

The pattern is identical in all of them. The engineering question looks like "remove duplicates" and the real question is "what makes two of these the same, and what do we lose when we are wrong".

That question has no technically correct answer. It has a correct answer for your product, which somebody has to make explicit, and the failure mode is not making it at all and inheriting whatever the first implementation happened to compare.

The honest limit

Normalisation makes deduplication possible. It does not make identity correct, and it introduces a risk of its own: the canonical shape is lossy by construction, because it keeps the fields the internal model has and discards the rest.

If a provider supplies something your shape has no room for, it is gone after the adapter, and you will not notice because nothing downstream ever knew it existed. That is an argument for keeping the raw payload alongside the canonical record, at least for a window, so a question about a field you did not model is answerable rather than unanswerable.

FAQ

Why can't I deduplicate results before normalising them? Because records in different shapes are not comparable at all. Field names, identifiers, units, precision and optionality all differ between sources, so there is no shared field to compare. Deduplication is structurally downstream of translating every source into one internal representation.

What decides whether two records are the same thing? A product decision, not a property of the data. Nothing in the records answers whether two itineraries with the same schedule but different fare rules are one result or two. The choice has to be made explicitly, and the common failure is inheriting whatever the first implementation happened to compare.

Which is worse, leaving duplicates or merging too aggressively? Merging too aggressively. A visible duplicate reads as a defect and the user can see both options. An over-broad merge removes an option the user would have chosen and leaves no trace in the output that anything was removed.

Which record should survive a merge? Ideally none of them alone. Build a canonical record taking the best-supported value per field, with provenance per field, rather than picking a winner. Keeping the cheapest fails when it is not bookable, keeping the first makes output depend on provider latency, and keeping the most complete rewards verbosity over accuracy.

How do I know if my deduplication is merging things it should not? Periodically keep both records instead of merging, compare them on the fields your identity rule ignores, and count how often they differed in ways a user would care about. Without that number, broadening identity always looks like progress because the duplicate rate falls monotonically.

What does normalisation lose? Anything the canonical shape has no room for. A field a provider supplies that your internal model does not represent disappears after the adapter, and nothing downstream knows it existed. Keeping the raw payload alongside the canonical record for a window makes questions about unmodelled fields answerable.