Deduplicating records received from multiple data sources

A record processing and storage system comprises:

• at least one processor;
• and
• a memory that stores operational instructions that, when executed by the at least one processor, cause the record processing and storage system to:
• receive, from a plurality of data sources via a communication network, a plurality of records streams that collectively include a plurality of records, wherein each of the plurality of record streams includes a corresponding set of records generated via a corresponding one of the plurality of data sources;
• generate a plurality of pages from the plurality of records included in the plurality of record streams, wherein corresponding page metadata is generated for each page in the plurality of pages based on a subset of the plurality of records included in the each page;
• store the plurality of pages in a page storage system;
• and
• facilitate deduplication of the plurality of pages based on a plurality of pages metadata of the plurality of pages based on, for the each page of the plurality of pages:
• identifying a filtered set of potentially-intersecting pages as a proper subset of the plurality of pages stored in the page storage system by comparing, in accordance with first comparison parameters, the corresponding page metadata of the each page with other page metadata of other ones of the plurality of pages stored in the page storage system;
• and
• identifying an intersecting set of pages that include a row number intersection with the each page as a proper subset of the filtered set of potentially-intersecting pages by comparing, in accordance with second comparison parameters, the corresponding page metadata of the each page with only the other page metadata of ones of the plurality of pages included in the filtered set of potentially-intersecting pages;
• and
• removing, from the each page, records with row numbers included in the row number intersection with each other page in the intersecting set of pages.

A non-transitory computer readable storage medium comprises:

• at least one memory section that stores operational instructions that, when executed by a processing module that includes a processor and a memory, cause the processing module to:
• receive, from a plurality of data sources via a communication network, a plurality of records streams that collectively include a plurality of records, wherein each of the plurality of record streams includes a corresponding set of records generated via a corresponding one of the plurality of data sources;
• generate a plurality of pages from the plurality of records included in the plurality of record streams, wherein corresponding page metadata is generated for each page in the plurality of pages based on a subset of the plurality of records included in the each page;
• store the plurality of pages in a page storage system;
• and
• facilitate deduplication of the plurality of pages based on a plurality of pages metadata of the plurality of pages based on, for the each page of the plurality of pages:
• identifying a filtered set of potentially-intersecting pages as a proper subset of the plurality of pages stored in the page storage system by comparing, in accordance with first comparison parameters, the corresponding page metadata of the each page with other page metadata of other ones of the plurality of pages stored in the page storage system;
• and
• identifying an intersecting set of pages that include a row number intersection with the each page as a proper subset of the filtered set of potentially-intersecting pages by comparing, in accordance with second comparison parameters, the corresponding page metadata of the each page with only the other page metadata of ones of the plurality of pages included in the filtered set of potentially-intersecting pages;
• and
• removing, from the each page, records with row numbers included in the row number intersection with each other page in the intersecting set of pages.