US 12,705,154 B2 · Real-time anomaly prediction using extrapolated telemetry data

Abstract

Systems and methods are disclosed for real-time anomaly prediction using near real-time data. The invention addresses delays in telemetry data collection from infrastructure components, by collecting metrics and logging this data in real-time. Extracted logged data undergoes initial analysis to identify patterns and anomalies, followed by cleaning to remove noise and errors. Feature engineering enhances the data, creating or modifying features to improve machine learning model performance. The system calculates weighted means of previous data values and computes first and second-order differences to capture immediate changes and trends. These calculations adjust the extrapolated value to accurately reflect current conditions. The adjusted data is integrated into the dataset and validated. The validated data trains and tests a machine learning model, which is then finalized and deployed for real-time anomaly detection. This system ensures accurate and timely anomaly prediction, enabling automated incident response to maintain the reliability and performance of infrastructure components.

A method for real-time anomaly prediction comprising the steps of:

collecting, by a sensor, telemetry data from various infrastructure components, including firewalls, switches, servers, and storage devices, capturing metrics including CPU usage, memory consumption, network traffic, and error rates;
logging, by a logging system, the collected telemetry data in real-time to create a continuous record of the telemetry data;
extracting, by a data extraction module, the logged telemetry data from the real-time log for further analysis and processing;
analyzing, by a data analysis module, the extracted telemetry data to identify patterns, trends, and any existing anomalies in the data;
cleaning, by a data cleaning module, the telemetry data to remove noise, irrelevant information, and errors, including handling missing values, smoothing outliers, and normalizing data;
performing feature engineering, by a feature engineering module, on the cleaned telemetry data to create or modify features that enhance the performance of a machine learning model, including calculating moving averages, trend lines, or interaction terms between different metrics;
calculating, by a calculation module, a weighted mean of a plurality of previous telemetry data values, wherein weights are the inverse of a distance from current time, to provide a baseline value for extrapolation;
computing, by a computation module, first-order differences between consecutive previous telemetry data values and the weighted mean to determine variation from the previous value and its own trends, wherein the weights are the inverse of the distance from the current time, capturing immediate changes in the data;
computing, by the computation module, second-order differences between consecutive previous telemetry data values and the weighted mean to determine the variation in the rate of change and its own trends, wherein the weights are the inverse of the distance from the current time, capturing acceleration or deceleration in the data trends;
adjusting, by an adjustment module, the extrapolated value for the current time based on the first-order and second-order differences to fine-tune the predicted value, ensuring it accurately reflects a current state of the infrastructure components;
integrating, by a data integration module, the adjusted extrapolated value into a dataset to create a continuous telemetry data stream, maintaining the integrity and continuity of the data for further analysis;
validating, by a validation module, extrapolated telemetry data by comparing it with actual telemetry data once it becomes available, assessing accuracy of the extrapolation process and making necessary adjustments to improve future predictions;
training, by a machine learning model, the machine learning model using the validated telemetry data to learn underlying patterns and improve its predictive capabilities;
testing, by a testing module, the trained machine learning model to evaluate its performance using a separate validation dataset, ensuring the model generalizes well to new, unseen data;
assessing, by an accuracy assessment module, the accuracy of the machine learning model using metrics such as precision, recall, F1 score, or mean absolute error to determine the model's performance and reliability;
adjusting, by a hyperparameter adjustment module, hyperparameters of the machine learning model if a desired accuracy is not achieved, optimizing the model's performance through iterative fine-tuning of parameters such as learning rate, number of layers, or number of trees in a random forest;
tuning, by a data tuning module, the telemetry data to improve model performance, including feature selection, feature scaling, and creating synthetic data to balance the dataset;
finalizing, by a model finalization module, the machine learning model once the desired accuracy is achieved, fixing model parameters and preparing it for deployment;
and
classifying, by the finalized machine learning model, anomalies in the telemetry data to predict whether a given data point represents a normal state or an anomaly, enabling timely detection and response to potential issues.

17 more elements

Real-time anomaly prediction using extrapolated telemetry data

Life of this patent

Real-time anomaly prediction using extrapolated telemetry data

Life of this patent

Abstract

Claims20