US 2008/0288482 A1 · Leveraging constraints for deduplication

Abstract

A deduplication algorithm that provides improved accuracy in data deduplication by using aggregate and/or groupwise constraints. Deduplication is accomplished using only as many of these constraints that are satisfied rather than be imposed inflexibly as hard constraints. Additionally, textual similarity between tuples is leveraged to restrict the search space. The algorithm begins with a coarse initial partition of data records and continues by raising the similarity threshold until the threshold splits a given partition. This sequence of splits defines a rich space of alternatives. Over this space, an algorithm finds a partition of the input that maximizes constraint satisfaction. In the context of groupwise aggregation constraints for deduplication all SQL (structured query language) aggregates are allowed, including summation.

Claims20

Dependent claims

A computer-implemented system for data deduplication, comprising:

a constraint component for providing a class of aggregate constraints for data processing;
and
a deduplication component for receiving an input relation of data records and outputting a space of partitions of the records based on the class of aggregate constraints.

Dependencies are inferred from the claim text.

Leveraging constraints for deduplication

Life of this patent

Leveraging constraints for deduplication

Life of this patent

Abstract

Claims20