Streaming and filtering event objects into a data lake

A computer program product comprising computer executable code embodied in a non-transitory computer readable medium that, when executing on one or more computing devices, causes the one or more computing devices to perform the steps of:

• detecting a plurality of security events with a local security agent executing on a compute instance in an enterprise network;
• transmitting the plurality of security events to an event stream for storage in a data lake of a threat management facility for the enterprise network;
• receiving a folding rule at the local security agent from the threat management facility, the folding rule based on the plurality of security events, and the folding rule executable by the local security agent to combine a characteristic sequence of security events in the plurality of security events into an aggregated security event, and to augment the aggregated security event with descriptive metadata;
• applying the folding rule on the compute instance to create an instance of the aggregated security event;
• and
• transmitting the instance of the aggregated security event to the threat management facility for the storage in the data lake.

A system comprising:

• a plurality of compute instances associated with an enterprise network;
• a data lake including a data storage medium storing a first plurality of data objects representing security events within the enterprise network and a plurality of descriptions for the first plurality of data objects, each of the plurality of descriptions organized according to one or more schemas;
• a stream service implemented by at least one processor to receive an asynchronous event stream of additional data objects representing the security events from the plurality of compute instances associated with the enterprise network;
• a transformer service implemented by the at least one processor, the transformer service configured to:
• process the asynchronous event stream by filtering the additional data objects to remove duplicate one or more data objects already stored in the data lake, thereby providing filtered data objects,
• augment each of the filtered data objects with a corresponding description organized according to one of the one or more schemas, thereby providing augmented data objects,
• store the augmented data objects in the data lake, and
• create a folding rule for aggregating a sequence of similar events into an aggregated security event for storage in the data lake, and for augmenting the aggregated security event with metadata that characterizes one or more of a number of events in the sequence of similar events;
• and
• a threat management facility configured to transmit the folding rule to the plurality of compute instances for use in monitoring activity in the enterprise network.