Filtering data logs from disparate data sources for processing by a security engine

A non-transitory computer-readable storage medium storing instructions for generating security alerts, the instructions when executed by a processor causing the processor to:

• obtain an input message stream from a set of data sources, wherein the input message stream comprises a sequence of input security-related data entries in different respective native formats that varies between different data sources of the set of data sources;
• identify a data format of a current input message of the input message stream using a set of detection rules, the current input message received from a current data source;
• parse the current input message, based on the identified data format of the current input message, to generate parsed message content, wherein the parsing includes separating the current input message into key-value pairs;
• apply a set of filter selection rules to the parsed message content to identify a filter from the set of filters for use with messages received from the current data source, wherein the filter selection rules identify a pattern associated with the current data source to identify the filter from the set of filters, and wherein each filter of the set of filters corresponds with one of the native formats of the different data sources of the set of data sources;
• apply the identified filter to the parsed message content to transform parsed message content to a current output message of an output message stream, wherein each output message of the output message stream comprises output security-related data entries in a standardized format independent of the data source;
• apply a set of security rules to the output message stream to detect a data pattern indicative of malicious activity;
• and
• generate a security alert to an administrative interface indicative of the detected data pattern.

A computer system comprising:

• a processor;
• and
• a non-transitory computer-readable storage medium storing instructions to generate security alerts, the instructions when executed by the processor to cause the processor to:
• obtain an input message stream from a set of data sources, wherein the input message stream comprises a sequence of input security-related data entries in different respective native formats that varies between different data sources of the set of data sources;
• identify a data format of a current input message of the input message stream using a set of detection rules, the current input message received from a current data source;
• parse the current input message, based on the identified data format of the current input message, to generate parsed message content, wherein the parsing includes separating the current input message into key-value pairs;
• apply a set of filter selection rules to the parsed message content to identify a filter from the set of filters for use with messages received from the current data source, wherein the filter selection rules identify a pattern associated with the current data source to identify the filter from the set of filters, and wherein each filter of the set of filters corresponds with one of the native formats of the different data sources of the set of data sources;
• apply, by the processor, the identified filter to the parsed message content to transform the parsed message content to a current output message of an output message stream, wherein each output message of the output message stream comprises output security-related data entries in a standardized format independent of the data source;
• apply a set of security rules to the output message stream to detect a data pattern indicative of malicious activity;
• and
• generate a security alert to an administrative interface indicative of the detected data pattern.