System and method for data ingestion, anomaly and root cause detection

A computing device, comprising:

• one or more streaming data processors;
• and
• a non-transitory storage medium having stored thereon an intake system being software including a plurality of components forming a data ingestion analytic subsystem, the data ingestion analytic subsystem comprises:
• a model training component configured to train (i) a first machine learning model of one or more machine learning models based on a first volume of data and (ii) a second machine learning model after determining a data ingestion volume is anomalous, and wherein the second machine learning model is trained on historical data sets consistent with features determined from an anomalous data ingestion volume as determined by deployment of the first machine learning model,
• a prediction component configured to conduct, using the second machine learning model, a predictive analysis of a sub-volume of the anomalous data ingestion volume to produce a predicted data ingestion sub-volume, wherein sub-volume is a portion of the anomalous data ingestion volume,
• an anomaly detection component configured to conduct a correlative analysis between the predicted data ingestion sub-volume and an actual data ingestion sub-volume to produce a prediction error, and
• an anomaly classification component configured to determine a degree of anomalousness of the prediction error, and
• a notification generation component configured to (i) generate a graphical user interface (GUI) providing a notification based on the degree of anomalousness associated with the prediction error and (ii) determine a cause of the anomalous data ingestion volume by at least (a) obtaining system state information during ingestion of the anomalous data ingestion sub-volume and (b) determining the cause of the anomalous data ingestion volume based on the system state information, wherein the GUI illustrates a graphical representation of the data ingestion volume over a prescribed time period and a visual representation of an error threshold relative to a predicted data ingestion volume over the prescribed time period, wherein a data ingestion anomaly is displayed as a visually distinct visual element from the graphical representation of the data ingestion volume and is displayed outside of the visual representation of the error threshold, and wherein the GUI displays a remedial recommendation for resolving the cause of the anomalous data ingestion value including an indication to fix or replace a component that is inoperable.

A non-transitory storage medium having stored thereon software that, when executed, is configured to perform operations comprising:

• detecting a data ingestion anomaly through deployment of a first machine learning model taking a data ingestion volume as input, wherein detection of the data ingestion anomaly indicates that the data ingestion volume is an anomalous data ingestion volume;
• determining features of the anomalous data ingestion volume;
• training a second machine learning model with historical data sets consistent with the determined features of the anomalous data ingestion volume;
• deploying a second machine learning model taking a data ingestion sub-volume as input, wherein the second machine learning model is configured to predict whether the data ingestion sub-volume is anomalous, wherein the data ingestion sub-volume is a portion of the anomalous data ingestion volume;
• obtaining system state information during ingestion of the anomalous data ingestion sub-volume;
• determining the cause of the anomalous data ingestion volume based on the system state information;
• and
• generating a graphical user interface (GUI) that illustrates a graphical representation of the data ingestion volume over a prescribed time period and a visual representation of an error threshold relative to a predicted data ingestion volume over the prescribed time period, wherein the data ingestion anomaly is displayed as a visually distinct visual element from the graphical representation of the data ingestion volume and is displayed outside of the visual representation of the error threshold, and wherein the GUI displays a remedial recommendation for resolving the cause of the anomalous data ingestion value including an indication to fix or replace a component that is inoperable.