Real time system for ingestion, aggregation, and identity association of data from user actions performed on websites or applications

A non-transitory computer-readable medium with instructions stored thereon that, responsive to execution by a processing device, cause the processing device to perform or control performance of operations, the operations comprising:

• storing a plurality of event records corresponding to a respective plurality of user interactions, the storing comprising:
• receiving the plurality of event records;
• compressing the plurality of event records as a compressed set of event records, stored in association with a file path;
• and
• writing the file path to a publish-subscribe queue;
• ingesting, from the publish-subscribe queue, a stream of data comprising the compressed set of event records, wherein the compressed set of event records is accessed using the file path, wherein the ingesting decompresses the compressed set of event records back into the plurality of event records and is performed in parallel to the storing;
• determining that a first record from the plurality of event records includes a first anonymous identifier and a first known identifier;
• adding a mapping between the first anonymous identifier and the first known identifier to an identifier resolution database;
• determining that a second record from the plurality of event records includes a second anonymous identifier and no known identifier;
• using the identifier resolution database to identify a second known identifier that is mapped to the second anonymous identifier;
• updating the second record to include the second known identifier;
• and
• including the first record and the updated second record in training data for training a machine learning model to compute causal inferences associated with the respective plurality of user interactions.

A system comprising:

• a memory with instructions stored thereon;
• and
• a processing device, coupled to the memory, the processing device configured to access the memory and execute the instructions, wherein the instructions cause the processing device to perform or control performance of operations, the operations comprising:
• storing a plurality of event records corresponding to a respective plurality of user interactions, the storing comprising:
• receiving the plurality of event records;
• compressing the plurality of event records as a compressed set of event records, stored in association with a file path;
• and
• writing the file path to a publish-subscribe queue;
• ingesting, from the publish-subscribe queue, a stream of data comprising the compressed set of event records, wherein the compressed set of event records is accessed using the file path, wherein the ingesting decompresses the compressed set of event records back into the plurality of event records and is performed in parallel to the storing;
• determining that a first record from the plurality of event records includes a first anonymous identifier and a first known identifier;
• adding a mapping between the first anonymous identifier and the first known identifier to an identifier resolution database;
• determining that a second record from the plurality of event records includes a second anonymous identifier and no known identifier;
• using the identifier resolution database to identify a second known identifier that is mapped to the second anonymous identifier;
• updating the second record to include the second known identifier;
• and
• including the first record and the updated second record in training data for training a machine learning model to compute causal inferences associated with the respective plurality of user interactions.