US 8,554,742 B2 · System and process for record duplication analysis

Abstract

A system and process for record duplication analysis that relies on a multi-membership Bayesian analysis to determine the probability that records within a data set are matches. The Bayesian calculation may rely on objective data describing the data set as well as subjective assessments of the data set. In addition, a system and process for record duplication analysis may rely on the predetermination of probabilistic patterns, where the system only searches for patterns exceeding a chosen threshold. Work flow may include selecting which fields within each record should be analyzed, normalizing the values within those fields and removing default data, calculating possible patterns and their match probabilities, analyzing record pairs to determine which have patterns exceeding a chosen threshold to determine the presence of duplicates, and merging duplicates, closing transactions reflecting non-duplicates, identifying records having insufficient data to determine the existence or lack of a match, and/or rolling back accidental merges.

Claims22

Dependent claims

A process operable on one or more computers for estimating a posterior probability of a match between a plurality of records in a database, each record having a plurality of fields, comprising:

calculating objective values relating to said fields;
approximating subjective values relating to said fields;
estimating an average number of duplicate records;
estimating a total number of distinct duplicate records using said subjective values;
determining a probability of a match between said records using said estimated average number of duplicate records and said estimated total number of distinct duplicate records;
determining a probability of no match between said records;
determining a conditional probability of a match between at least one of said fields of said records, given a match between said records, using said subjective values;
determining a conditional probability of a match between at least one of said fields of said records, given no match between said records, using both said objective values and said subjective values;
and
using a Bayesian probability function to determine said posterior probability based on said probabilities and said conditional probabilities.

7 more elements

A process operable on one or more computers for identifying duplicate records among a plurality of records in a database, each record having a plurality of fields, comprising:

(a) setting a threshold match probability;
(b) selecting one or more of said fields for comparison;
(c) calculating record match probabilities for each of a plurality of possible patterns, wherein each pattern is a different permutation of comparisons between said fields and wherein each record match probability is a posterior probability that two records match given that the two records fit the respective pattern;
(d) identifying patterns having record match probabilities higher than said threshold match probability;
(e) disregarding patterns having record match probabilities lower than said threshold match probability;
(f) determining which record pairs within the plurality of records have one or more of said identified patterns;
and
(g) analyzing said record pairs to determine whether said records are duplicates;
and
(h) merging duplicate records,
wherein said threshold match probability and said record match probabilities are Bayesian posterior probabilities;
and
wherein steps (c)-(e) occur prior to steps (f)-(h).

10 more elements

Dependencies are inferred from the claim text.

System and process for record duplication analysis

Life of this patent

System and process for record duplication analysis

Life of this patent

Abstract

Claims22