Detecting duplicate records in databases

In a database having records stored on a medium, a method for identifying possible duplicate records comprising:

• a) evaluating a first containment metric for tokens contained in a first field of multiple records from one or more database tables to select a first set of candidate records as possible duplicate records;
• b) evaluating a second containment metric for tokens contained in a second field from the one or more database tables that is hierarchically connected to the first set of candidate records;
• and
• c) identifying two or more records within the one or more tables as possible duplicate records by using the first and second containment metrics to produce an output in the event a duplicate record threshold is met.

In a database having records stored on a medium, a method for identifying possible duplicate records comprising:

• a) providing multiple tables that are related by key-foreign key relations between tables to form a hierarchy of records;
• b) identifying duplicate contents within at least one field within the records of a first table;
• c) grouping the records of a second table related to said first table based on the identification of the duplicate fields in the first table;
• and
• d) identifying duplicate record pairs from the first and second tables based on a search for duplicate contents of fields in the records of the second table that were grouped based on duplicate contents of fields from records in the first table.

A system for evaluating records in a database to determine a presence of duplicate records comprising:

• a) one or more computers for storing data that is organized according to a hierarchy of related fields in one or more database tables;
• and
• b) a database management system including a processor for selectively extracting records from the one or more database tables and including processor components for evaluating the contents of said records;
• c) said processor including a duplicate determination component that i) accesses multiple records in one or more database tables that include multiple fields;
• and ii) identifies two or more records within a table as duplicates by measuring a co-occurence of data in hierarchically related fields of the one or more tables.