Dynamic record blocking
AssigneeIntelius
InventorsWilliam P. McNeill, Andrew Borthwick
Life of this patent
Estimating the term…
Abstract
Dynamic blocking determines which pairs of records in a data set should be examined as potential duplicates. Records are grouped together into blocks by shared properties that are indicators of duplication. Blocks that are too large to be efficiently processed are further subdivided by other properties chosen in a data-driven way. We demonstrate the viability of this algorithm for large data sets. We have scaled this system up to work on billions of records on an 80 node Hadoop cluster.
Claims22
A method of using a plurality of distributed computing nodes each comprising a processor and associated storage to dynamically block records, comprising:
- (a) using the distributed computing nodes comprising a processor and associated storage, grouping together records of a data set with a first set of shared properties into blocks;
- and
- (b) using the distributed computing nodes comprising a processor and associated storage, for a block that is intractably large and therefore requires further grouping records into sub-blocks, automatically discovering based on the contents of the intractably large block, at least one second set of shared properties that enables creation of sub-blocks of tractable size, wherein the second set is different from the first set.
A system for dynamically blocking records, comprising:
- a plurality of distributed computing nodes each comprising a processor and associated non-transitory memory, the nodes each storing in non-transitory storage codes that when executed in parallel:
- groups together records of a data set with a first set of similar properties into blocks;
- and
- for a block that is intractably large and therefore requires further grouping records into sub-blocks, automatically discovers based on the contents of the intractably large block, at least one second set of shared properties that enables creation of sub-blocks of tractable size, wherein the second set is different from the first set.
Dependencies are inferred from the claim text.