Recurrent neural network based anomaly detection

A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to specifically configure the data processing system to:

• receive, from a computing system resource of the monitored computing environment, a system call of an observed system call sequence for evaluation;
• process, by a trained recurrent neural network (RNN) of the data processing system, trained to predict system call sequences, the system call to generate a prediction of a subsequent system call in a predicted system call sequence;
• compare, by abnormal call sequence logic of the data processing system, the subsequent system calls in the predicted system call sequence to an observed system call in the observed system call sequence;
• identify, by the abnormal call sequence logic, a difference between the predicted system call sequence and the observed system call sequence based on results of the comparing;
• and
• generate, by the abnormal call sequence logic, an alert notification in response to identifying the difference, wherein the prediction of the subsequent system call is generated at least by:
• generating a plurality of combinations of system call features and argument features, from a plurality of system call features and plurality of argument features of the system call, and for each combination in the plurality of combinations, generating a probability for the combination;
• and
• selecting a combination from the plurality of combinations to represent the predicted subsequent system call based on the probabilities for the combinations in the plurality of combinations.

A data processing system comprising:

• a recurrent neural network (RNN);
• and
• a processor configured to execute abnormal call sequence logic, wherein:
• the RNN is trained to predict system call sequences, the RNN receives, from a computing system resource of the monitored computing environment, at least one system call of an observed system call sequence for evaluation;
• the RNN processes the at least one system call to generate a prediction of a subsequent system call in a predicted system call sequence;
• the abnormal call sequence logic compares the subsequent system call in the predicted system call sequence to an observed system call in the observed system call sequence;
• the abnormal call sequence logic identifies a difference between the predicted system call sequence and the observed system call sequence based on results of the comparing;
• and
• the abnormal call sequence logic generates an alert notification in response to identifying the difference, wherein the prediction of the subsequent system call is generated at least by:
• generating a plurality of combinations of system call features and argument features, from a plurality of system call features and plurality of argument features of the system call, and for each combination in the plurality of combinations, generating a probability for the combination;
• and
• selecting a combination from the plurality of combinations to represent the predicted subsequent system call based on the probabilities for the combinations in the plurality of combinations.