Method and system for deduplicating data

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

• receiving the set of records by receiving a data table from storage, wherein the data table comprises one or more columns each represent a respective attribute and one or more rows each representing an attribute value;
• identify a subset of records of the set of records each having one or more invalid attribute values by retrieving heuristic rules to determine a likelihood that an attribute value is an invalid value and applying the heuristic rules to each of the attribute values;
• determine for each invalid attribute value of a given attribute of the subset of the set of records one or more associated valid candidates of attribute values of the given attribute using the set of records, wherein determining for a given invalid attribute value one or more associated valid candidates of attribute values comprises selecting records of the set of records having a same super key as the record comprising the given invalid attribute value and determining the one or more associated valid candidates of the attribute values using the selected records;
• for each record of the subset of records of the set of records:
• replace the one or more invalid attribute values by one or more combinations of the determined valid candidates of attribute values resulting in a modified set of records;
• select a subset of records of the modified set of records that satisfy a consistency condition on the attribute values of each record, wherein the consistency condition comprises a rule for determining whether the attribute values of each record are consistent;
• deduplicate, in the data table in the storage, the subset of records of the modified set of records responsive to determining the subset of records of modified set of records comprises more than one record.