System and method for automatically generating neural networks for anomaly detection in log data from distributed systems

A non-transitory computer-readable storage medium containing program instructions for method for automatically generating recurrent neural networks for log anomaly detection, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to perform steps comprising:

• creating a training dataset of logs and a validation dataset of logs, wherein the logs in the training and validation datasets relate to operations of at least one component running in a computing environment;
• for each input set of controller parameters that is applied to a controller recurrent neural network:
• generating an output set of hyperparameters at the controller recurrent neural network;
• applying the output set of hyperparameters to a target recurrent neural network to produce a child recurrent neural network for log anomaly detection with an architecture that is defined by the output set of hyperparameters;
• training the child recurrent neural network using the training dataset of logs to classify each of the logs as one of an anomalous log and a non-anomalous log;
• and
• computing a log classification accuracy of the child recurrent neural network with respect to correct classification of anomalous logs and non-anomalous logs using the validation dataset of logs;
• and
• using a current log classification accuracy of a corresponding child recurrent neural network, adjusting at least one of the controller parameters used to generate the corresponding child recurrent neural network to produce a different input set of controller parameters to be applied to the controller recurrent neural network so that a different child recurrent neural network for log anomaly detection with a different architecture can be generated.

A system for automatically generating recurrent neural networks for log anomaly detection, the system comprising:

• memory;
• and
• at least one processor configured to:
• create a training dataset of logs and a validation dataset of logs, wherein the logs in the training and validation datasets relate to operations of at least one component running in a computing environment;
• for each input set of controller parameters that is applied to a controller recurrent neural network:
• generate an output set of hyperparameters at the controller recurrent neural network;
• apply the output set of hyperparameters to a target recurrent neural network to produce a child recurrent neural network for log anomaly detection with an architecture that is defined by the output set of hyperparameters;
• train the child recurrent neural network using the training dataset of logs to classify each of the logs as one of an anomalous log and a non-anomalous log;
• and
• compute a log classification accuracy of the child recurrent neural network with respect to correct classification of anomalous logs and non-anomalous logs using the validation dataset of logs;
• and
• using a current log classification accuracy of a corresponding child recurrent neural network, adjust at least one of the controller parameters used to generate the corresponding child recurrent neural network to produce a different input set of controller parameters to be applied to the controller recurrent neural network so that a different child recurrent neural network for log anomaly detection with a different architecture can be generated.