Automatically change anomaly detection threshold based on probabilistic distribution of anomaly scores

#The present invention relates to model decay of an anomaly detector due to concept drift. Herein are machine learning techniques for dynamically self-tuning an anomaly score threshold.

#Anomaly detection is an important tool with various use cases in security such as fraud detection and intrusion detection. A powerful approach recently employed in anomaly detection involves machine learning (ML) models such as an artificial neural network such as an autoencoder. Anomaly detection using ML models can be categorized into two groups of existing techniques. In the first group, the entire dataset is used for model training, and the ML model is used to detect anomalies inside that dataset. In the second group, the data is discretized into three parts: training, validation, and test. The ML model is trained on the training dataset and then fine-tuned using the validation dataset. Subsequently the trained and validated model is applied to the test dataset. The test dataset can be a nearly infinite stream of data incoming to the ML model. Although such models can be made somewhat robust to input noise, they cannot handle input data distribution changes, which is a phenomenon called concept drift or data drift.

#In one example, concept drift occurs when a series of inputs evolves over time such that later normal inputs have feature values that are no longer within the value range of earlier normal inputs such as training inputs. For example, an ML model may be a classifier that learned to distinguish normal tree seedlings from normal weeds because seedlings are smaller than weeds. However over time, seedlings may grow to be a same size as weeds or grow into trees that are bigger than weeds, which may confuse the classifier. For example, the classifier may mistakenly classify an older seedling as a weed.

#In the drawings:

#FIG. 1 is a block diagram that depicts an example computer that dynamically tunes an adaptive anomaly threshold to compensate for concept drift that could cause model decay of a machine learning (ML) model;

#FIG. 2 is a flow diagram that depicts an example computer process that continuously tunes an adaptive anomaly threshold to compensate for concept drift that could cause model decay of a machine learning (ML) model;

#In the following description, for the purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It will be apparent, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to avoid unnecessarily obscuring the present invention.

#The present invention relates to model decay of an anomaly detector due to concept drift. Herein are machine learning techniques for dynamically self-tuning an anomaly score threshold. Herein is a dynamic thresholding approach to preserve an anomaly detector's ability to accurately classify data items even when concept drift causes a distribution of data in a production environment to diverge from training data. A goal is to make the anomaly detector able to adapt to changes in value range and distribution trends of input data. Herein is anomaly score normalization and dynamic thresholding that are proven to be effective in detecting and automatically handling concept drifts in data of various applications such as structured log data of cloud servers.

#Compared to other industry solutions, approaches herein allow a machine learning (ML) anomaly detector to continue to perform well for longer periods of time and with no need to be retrained on recent data. Techniques herein are computationally inexpensive and can be done in streaming mode with no need to have all data stored in memory, which may be infeasible for large-scale data.