Methods and systems for server failure prediction using server logs

An anomaly prediction system, comprising:

• a communication interface;
• a memory comprising executable instructions;
• and
• a processor communicably coupled to the communication interface, the processor configured to execute the executable instructions to cause the anomaly prediction system to at least:
• access a set of distinct log clusters associated with a server, the set of distinct log clusters representing instructions executed in the server;
• configure a first density machine learning model based on a stacked autoencoder configured with a loss function that includes a Log-Cosh component and a constraint on a summation of frequency appearances of the set of distinct log clusters, the loss function structured to enable the autoencoder to reconstruct an input vector even when some log clusters are less frequent;
• apply the first density machine learning model over the input vector associated with the set of distinct log clusters for obtaining a first prediction output, the input vector having a length equal to a number of the set of distinct log clusters, the first density machine learning model trained based on normal log data associated with the server, to detect a first deviation of an input vector from an expected density distribution of log data that corresponds to normal operation, and generating the first prediction output based on the first deviation;
• apply a first sequential machine learning model over a time length sequence of the set of distinct log clusters for obtaining a second prediction output, the first sequential machine learning model trained based on the normal log data associated with the server, to detect a second deviation of the input vector from an expected sequence of the log data that corresponds to normal operation, and generating the second prediction output based on the second deviation;
• wherein the first sequential machine learning model trained on the normal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
• apply a second density machine learning model over the input vector for obtaining a third prediction output, the second density machine learning model trained based on abnormal log data associated with the server, to detect a third deviation of the input vector from an expected density distribution of the log data that corresponds to abnormal operation, and generating the third prediction output based on the third deviation, apply a second sequential machine learning model over the time length sequence of the set of distinct log clusters for obtaining a fourth prediction output, the second sequential machine learning model trained based on the abnormal log data associated with the server, to detect a fourth deviation of the input vector from an expected sequence of the log data that corresponds to abnormal operation, and generating the fourth prediction output based on the fourth deviation;
• wherein the second density machine learning model trained on the abnormal log data of the server is implemented using a stacked auto encoder, and the second sequential machine learning model trained on the abnormal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
• update, dynamically during a detection phase based on feedback indicating a misclassification, a probability weight distribution of:
• the first density machine learning model, the first sequential machine learning model, the second density machine learning model, and/or the second sequential machine learning model, thereby adapting to new log patterns to improve respective model performance;
• aggregate the first, second, third and fourth prediction outputs based on an ensemble model;
• and
• predict a likelihood of next log clusters to have anomalous behavior based on the aggregating.

A computer-implemented method, comprising:

• accessing, by a processor, a set of distinct log clusters associated with a server, the set of distinct log clusters representing instructions executed in the server;
• configuring, by the processor, a first density machine learning model based on a stacked autoencoder configured with a loss function that includes a Log-Cosh component and a constraint on a summation of frequency appearances of the set of distinct log clusters, the loss function structured to enable the autoencoder to reconstruct an input vector even when some log clusters are less frequent;
• applying, by the processor, the first density machine learning model over the input vector associated with the set of distinct log clusters for obtaining a first prediction output, the input vector having a length equal to a number of the set of distinct log clusters, the first density machine learning model trained based on normal log data associated with the server to detect a first deviation of an input vector from an expected density distribution of log data that corresponds to normal operation, and generating the first prediction output based on the first deviation;
• applying, by the processor, a first sequential machine learning model over a time length sequence of the set of distinct log clusters for obtaining a second prediction output, the first sequential machine learning model trained based on the normal log data associated with the server to detect a second deviation of the input vector from an expected sequence of the log data that corresponds to normal operation, and generating the second prediction output based on the second deviation;
• wherein the first sequential machine learning model trained on the normal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
• generating, by the processor, an ensemble output of the first prediction output and the second prediction output for identifying whether server anomalies are detected or not;
• and in response to identification of the server anomalies, performing:
• applying, by the processor, a second density machine learning model over the input vector for obtaining a third prediction output, the second density machine learning model trained based on abnormal log data associated with the server to detect a third deviation of the input vector from an expected density distribution of the log data that corresponds to abnormal operation, and generating the third prediction output based on the third deviation;
• applying, by the processor, a second sequential machine learning model over the time length sequence of the set of distinct log clusters for obtaining a fourth prediction output, the second sequential machine learning model trained based on the abnormal log data associated with the server to detect a fourth deviation of the input vector from an expected sequence of the log data that corresponds to abnormal operation, and generating the fourth prediction output based on the fourth deviation;
• wherein the second density machine learning model trained on the abnormal log data of the server is implemented using a stacked auto encoder, and the second sequential machine learning model trained on the abnormal log data of the server is implemented using a Long Short Term Memory (LSTM) based sequential auto encoder;
• updating, by the processor, dynamically during a detection phase based on feedback indicating a misclassification, a probability weight distribution of:
• the first density machine learning model, the first sequential machine learning model, the second density machine learning model, and/or the second sequential machine learning model, thereby adapting to new log patterns to improve respective model performance;
• generating, by the processor, a second ensemble output of the third prediction output and the fourth prediction output based at least part on an ensemble model;
• predicting, by the processor, a likelihood of next log clusters to have anomalous behavior based on the second ensemble output.