Duplicate Record

How duplicates arise

Duplicates almost always arise wherever the same objects can be created in several places or by several people. The most common cause is manual re-entry without a prior search: an employee cannot find an existing customer – because they are spelled differently – and simply creates them anew. Typos, inconsistent spellings of company names, differing address formats, and missing mandatory fields also encourage duplicate entries.

A second major source is interfaces and imports. If customers from an online shop, a marketplace, and an Excel list are imported into the same ERP without a unique matching key, the same customer quickly ends up in the system multiple times. Data migration during a system change is particularly critical: when legacy data from several predecessor systems is merged, the duplicates maintained there collide and multiply. This is why the principle applies that master data should be cleaned before migration rather than carrying the problem along.

How a duplicate is detected and cleaned up

Because duplicates are rarely identical field by field, an exact comparison is not enough. Instead, fuzzy methods are used that measure similarity rather than equality. They first normalize the values (upper/lower case, legal-form abbreviations, special characters), then form comparison keys and evaluate the match across several fields together. Only a sufficiently high match score flags two records as a potential duplicate – for manual review or automatic merging.

Why a duplicate becomes expensive in the ERP

The damage caused by duplicates is rarely spectacular but broadly spread. On the sales side, a customer’s revenue is split across several records, so that ABC analyses, credit limits, and discount tiers are distorted – a genuinely important A customer appears as several insignificant B customers. Dunning runs come to nothing because open items are on one record and payments on another. In marketing, duplicates lead to multiple mailings, wasted postage, and annoyed recipients; under the GDPR they also make it harder to fully satisfy access and erasure requests.

With product duplicates, stock fragments: the same goods are recorded on two item numbers for accounting purposes, leading to incorrect availabilities, unnecessary reorders, and faulty contribution margins. Because errors in master data propagate into every downstream document, freedom from duplicates is one of the most important metrics of data quality and is a core objective of Master Data Management (MDM).

Distinctions: duplicate, homonym, and golden record

A duplicate denotes two records for one object and is thus the opposite of the homonym problem, in which one record wrongly merges two different real-world objects (for example two different companies under one collective record). Both are data quality errors but require opposite corrections: duplicates are merged, homonyms are split apart.

The duplicate differs from the golden record as cause differs from result: the duplicate is the faulty initial state, the golden record is the cleaned, leading target record. Not every similarity is a duplicate, either – two separate delivery addresses of the same customer or product variants with their own item numbers are intended by design and must not be merged. A good duplicate check therefore only proposes candidates; the business decision remains with the responsible data owner.

DACH specifics and data governance

In the German-speaking region, several factors come together that both encourage duplicates and make them sensitive. Legal-form suffixes such as GmbH, GmbH & Co. KG, or e. K. are written and abbreviated inconsistently, and umlauts are captured sometimes as "ä" and sometimes as "ae" – both classic triggers of undetected duplicate entries. A reliable unique key such as the VAT identification number in the customer and supplier master helps to recognize such records as the same partner despite differing spellings.

Legally, the GDPR sharpens the topic: personal-data duplicates not only complicate access and erasure, their avoidance is part of the principle of data accuracy. The problem is solved sustainably less through one-off actions than through structured data governance – mandatory fields, naming conventions, a duplicate check already at entry, and clear responsibilities. This keeps the data set permanently clean instead of having to be cleaned up again at regular intervals.