Load distribution enabling detection of first appearance of a new property value in pipeline data processing

A method of load distribution of events in an event stream by distributing events generated by a group into n parallel computational elements comprising:

• receiving by a processor of a computing device an event stream of events generated by a group comprising sub-groups;
• receiving a configurable time period which when exceeded triggers generation of a notification to the group;
• distributing the events between computational elements based on a hash of sub-group identifiers;
• extracting event data from events in the event stream, the events comprising a new property value, where a new property value is a value that has not been detected within the configurable time period;
• aggregating the extracted event data;
• determining the earliest detection for the group;
• sending updates to all the sub-groups of the groups;
• and
• generating a notification identifying the group, sub-group and property value.

A system that detects a potential security threat, the system comprising:

• a memory connected to a processor, the processor configured to detect a first appearance of a property value in an event stream for a group in a configurable time period by:
• accessing retained event data identifying a last time at which the property value was detected in an event stream for the group, the group comprising sub-groups;
• and
• in response to determining that a time period between the last time in the retained event data and a first new detection of the property value exceeds a configurable amount of time, triggering generation of a report comprising a potential security threat, the report identifying a group, a sub-group and a value of a property by assigning events of a sub-group of an event-generating group to one of n computational elements based on a hash value of the sub-group identifier.