Incremental data processing

A computer program product comprising a non-transitory computer-readable program medium having code stored thereon, the code when executed, causing one or more processors to implement a method comprising:

• receiving multiple event records including a first event record and a second event record, wherein the first event record comprises at least one of a first event identification, a first event generation time stamp or a first event description field, and the second event record comprises at least one of a second event identification, a second event generation time stamp or a second event description field;
• generating, for the first event record, a first salt value;
• generating, for the second event record, a second salt value;
• distributing, based on at least the first salt value and the second salt value, the multiple event records over storage locations;
• and
• generating unique events in an advertisement delivery network associated with the storage locations based on a comparison of a first number of events logged at an ad server with a second number of events generated for billing by a billing system.

A communication system, comprising:

• one or more processors that communicate event records over a transport network;
• and
• a de-duplication mechanism that receives the event records from the transport network, wherein the de-duplication mechanism includes:
• a randomizer module that randomizes the event records to generate randomized event records;
• a distribution module that distributes the randomized event records to multiple computer platforms;
• a chunking module that reads a subset of the event records on at least one of the multiple computer platforms;
• and
• a comparator module that identifies one or more duplicate entries based on unique event identifiers, wherein each unique event identifier comprises a salt value generated as a function of an event identification for a corresponding event record.