Automatic and scalable log pattern learning in security log analysis

A computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method for implementing automatic and scalable log pattern learning in security log analysis, comprising:

• collecting security logs generated by one or more management services of a computer system;
• implementing an incremental learning process to generate a set of log patterns from the collected security logs;
• and
• parsing the collected security logs using the set of log patterns;
• wherein implementing the incremental learning process to generate the set of log patterns further comprises:
• defining a first set as the training set, a second set as a set of log patterns that have been generated, and a third set as a set of logs of the training set that lack a matching pattern in the second set;
• sampling the third set to generate a fourth set having a size corresponding to a parameter controlling a maximum resource requirement for the incremental learning process;
• performing automatic log pattern recognition to generate a fifth set;
• and
• performing a log filtering process based on the fifth set.