Scaling record linkage via elimination of highly overlapped blocks

A computer-implemented method comprising:

• identifying a first plurality of blocks based at least on a plurality of records stored in a storage service of a provider network;
• identifying a plurality of sets of matching blocks from the first plurality of blocks;
• deleting the plurality of sets of matching blocks except for a first block from each set from the plurality of sets of matching blocks;
• and
• iteratively performing dynamic blocking based at least on the first block to generate subsequent pluralities of blocks until the subsequent pluralities of blocks are below a threshold size.

A system comprising:

• a storage service implemented by a first one or more electronic devices;
• and
• a data lake service implemented by a second one or more electronic devices, the data lake service including instructions that upon execution cause the data lake service to:
• identify a first plurality of blocks based at least on a plurality of records stored in a storage service of a provider network;
• identify a plurality of sets of matching blocks from the first plurality of blocks;
• delete the plurality of sets of matching blocks except for a first block from each set from the plurality of sets of matching blocks;
• and
• iteratively perform dynamic blocking based at least on the first block to generate subsequent pluralities of blocks until the subsequent pluralities of blocks are below a threshold size.