System and method for using probabilistic data structure to synchronize data between a plurality of datasets

A system for synchronizing data between a plurality of datasets, the system comprising:

• at least one memory having processor-readable instructions stored therein;
• and at least one processor configured to access the memory and execute the processor-readable instructions, which when executed by the processor configure the processor to perform a plurality of functions, including functions for:
• receiving, by at least one processor, a request to match a subject record in a first dataset with a corresponding record in a remote, second dataset based on at least one synchronization criteria;
• dividing, by at least one processor, the second dataset into a plurality of chunks each comprising a predetermined number of records;
• generating, by at least one processor, a probabilistic data structure respectively associated with a first of the plurality of chunks and configured to determine whether the corresponding record is contained in the first of the plurality of chunks, wherein the generating the probabilistic data structure associated with the first of the plurality of chunks comprises performing one or more hash functions on keys of records in the associated chunk to map the keys to a bit vector;
• determining, by at least one processor, within a predetermined level of accuracy, that the corresponding record is present in the first of the plurality of chunks based on the probabilistic data structure, wherein determining that the corresponding record exists in the first of the plurality of chunks comprises performing a hash function on a key of the subject record;
• and
• causing, by at least one processor, one or more attributes of the subject record to be synchronized with one or more attributes of the corresponding record based on existence of the corresponding record in the first of the plurality of chunks.

A non-transitory computer-readable medium containing instructions synchronizing data between a plurality of datasets, the non-transitory computer-readable medium storing instructions that, when executed by at least one processor, configure the at least one processor to perform:

• receiving, by at least one processor, a request to match a subject record in a first dataset with a corresponding record in a remote, second dataset based on at least one synchronization criteria;
• dividing, by at least one processor, the second dataset into a plurality of chunks each comprising a predetermined number of records;
• generating, by at least one processor, a probabilistic data structure respectively associated with a first of the plurality of chunks and configured to determine whether the corresponding record is contained in the first of the plurality of chunks, wherein the generating the probabilistic data structure associated with the first of the plurality of chunks comprises performing one or more hash functions on keys of records in the associated chunk to map the keys to a bit vector;
• determining, by at least one processor, within a predetermined level of accuracy, that the corresponding record is present in the first of the plurality of chunks based on the probabilistic data structure, wherein determining that the corresponding record exists in the first of the plurality of chunks comprises performing a hash function on a key of the subject record;
• and
• causing, by at least one processor, one or more attributes of the subject record to be synchronized with one or more attributes of the corresponding record based on existence of the corresponding record in the first of the plurality of chunks.