Real time processing for aggregates

A system for determining attribute aggregations, comprising:

• one or more processors;
• and
• at least one memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• receiving a stream of data over a communications network from one or more sources, the data comprising domain events associated with a plurality of primary keys, each domain event comprising one or more attributes;
• ingesting the data with a distributed ingestion service, wherein ingesting the data comprises:
• deduplicating the data to extract only a most recent domain event for each primary key;
• selecting for ingestion only fields in the most recent domain event for each primary key that are required for determining attribute aggregations;
• and
• writing selected fields from the most recent domain event for each primary key to multiple partitions in folders on a network storage device and uploading the selected fields from the most recent domain event for each primary key in a bulk import into a distributed database partitioned based on primary keys;
• and
• executing read queries by a distributed query service by reading data from the distributed database based on the primary keys and determining and publishing the attribute aggregations from the data in response to one or more queries.

A method for determining attribute aggregations in a stream processing system, the method comprising:

• receiving queries for attribute aggregations of data streamed over a communications network, wherein the data comprises domain events associated with a plurality of primary keys, each domain event comprising one or more attributes and is stored in distributed database partitioned based on primary keys;
• and
• executing the queries by a distributed query service comprising:
• reading the data from the distributed database based on the primary key;
• filtering the data based on a time of ingestion of the data to select only attributes updated within a user defined amount of time;
• deduplicating filtered data when multiple versions are present to extract only most recent attribute values for each primary key;
• determining the attribute aggregations for deduplicated and filtered data;
• and
• publishing the attribute aggregations in response to the queries.

A system for determining attribute aggregations, comprising:

• one or more processors;
• and
• at least one memory coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the system to perform operations comprising:
• receiving queries for attribute aggregations of data streamed over a communications network, wherein the data comprises domain events associated with a plurality of primary keys, each domain event comprising one or more attributes and is stored in distributed database partitioned based on primary keys;
• and
• executing the queries by a distributed query service comprising:
• reading the data from the distributed database based on the primary key;
• filtering the data based on a time of ingestion of the data to select only attributes updated within a user defined amount of time;
• deduplicating filtered data when multiple versions are present to extract only most recent attribute values for each primary key;
• determining the attribute aggregations for deduplicated and filtered data;
• and
• publishing the attribute aggregations in response to the queries.