Automatic entity resolution data cleaning

A computer program product, stored on a non-transitory computer readable medium, comprising instructions that when executed on one or more computers cause the one or more computers to perform operations implementing automatic entity resolution (ER) data cleaning, the operations comprising:

• receiving a group of N references to a candidate ER error entity,
• wherein the references and the candidate ER entity are persistent data stored in a data repository, and wherein the references each include attribute data describing the candidate ER entity;
• identifying a set of related references within the group of N references,
• wherein each related reference is related to at least one other reference of the group of N references;
• for each pair of references identified as a part of the set of the related references, calculating an ER score representing a likelihood that the pair of related references refers to the candidate ER error entity;
• determining based on the calculation of each ER score, that an ER error has occurred in grouping the references;
• selecting a set of the pairs of the related references for validation of their ER scores,
• sending the selected set of the pairs to an oracle for validation of their ER scores;
• receiving validated ER scores from the oracle, the validated ER scores indicative of a match or non-match and are associated with a value between 0 and 1;
• adjusting at least one of the validated ER scores by calculating a paralyzed ER score including determining whether the validated ER score satisfies a match threshold;
• and
• performing a recursive re-grouping process comprising:
• re-grouping, using a grouping method, the set of related references based in part on their respectively associated ER scores forming additional new input data based on the re-grouping of the references, the respectively associated ER scores comprising the paralyzed ER score;
• re-assigning the group of each of the N references based on the re-grouping;
• and
• recalculating a pairwise matching of the additional new input data formed based on the re-grouping of the references.

A method comprising:

• receiving a group of N references to a candidate ER error entity,
• wherein the references and the candidate ER entity are persistent data stored in a data repository, and wherein the references each include attribute data describing the candidate ER entity;
• identifying a set of related references within the group of N references,
• wherein each related reference is related to at least one other reference of the group of N references;
• for each pair of references identified as a part of the set of the related references, calculating an ER score representing a likelihood that the pair of related references refers to the candidate ER error entity;
• determining based on the calculation of each ER score, that an ER error has occurred in grouping the references;
• selecting a set of the pairs of the related references for validation of their ER scores,
• sending the selected set of the pairs to an oracle for validation of their ER scores;
• receiving validated ER scores from the oracle, the validated ER scores indicative of a match or non-match and are associated with a value between 0 and 1;
• adjusting at least one of the validated ER scores by calculating a paralyzed ER score including determining whether the validated ER score satisfies a match threshold;
• and
• performing a recursive re-grouping process comprising:
• re-grouping, using a grouping method, the set of related references based in part on their respectively associated ER scores forming additional new input data based on the re-grouping of the references, the respectively associated ER scores comprising the paralyzed ER score;
• re-assigning the group of each of the N references based on the re-grouping;
• and
• recalculating a pairwise matching of the additional new input data formed based on the re-grouping of the references.

A system comprising one or more computers and one or more non-transitory storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations implementing automatic entity resolution (ER) data cleaning, the operations comprising:

• receiving a group of N references to a candidate ER error entity,
• wherein the references and the candidate ER entity are persistent data stored in a data repository, and wherein the references each include attribute data describing the candidate ER entity;
• identifying a set of related references within the group of N references,
• wherein each related reference is related to at least one other reference of the group of N references;
• for each pair of references identified as a part of the set of the related references, calculating an ER score representing a likelihood that the pair of related references refers to the candidate ER error entity;
• determining based on the calculation of each ER score, that an ER error has occurred in grouping the references;
• selecting a set of the pairs of the related references for validation of their ER scores,
• sending the selected set of the pairs to an oracle for validation of their ER scores;
• receiving validated ER scores from the oracle, the validated ER scores indicative of a match or non-match and are associated with a value between 0 and 1;
• adjusting at least one of the validated ER scores by calculating a paralyzed ER score including determining whether the validated ER score satisfies a match threshold;
• and
• performing a recursive re-grouping process comprising:
• re-grouping, using a grouping method, the set of related references based in part on their respectively associated ER scores forming additional new input data based on the re-grouping of the references, the respectively associated ER scores comprising the paralyzed ER score;
• re-assigning the group of each of the N references based on the re-grouping;
• and
• recalculating a pairwise matching of the additional new input data formed based on the re-grouping of the references.