# US 12,216,527 B1 · System and method for data ingestion, anomaly and root cause detection > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,216,527 B1 - Document ID: US12216527B1 - Status: Granted - Kind: B1 - Application number: 17/583,056 - Filed: 2022-01-24 - Granted: 2025-02-04 - Earliest claimed: 2022-01-24 - Assignees: Splunk Inc. - Applicants: Splunk, Inc. - Inventors: Abraham Starosta; Francis Beckert; Chandrima Sarkar - Web page: https://patentgrep.com/patent/US12216527B1 ## CPC - G06F 11/0751: {Error or fault detection not based on redundancy (power supply failures G06F1/30; network fault management H04L41/06)} - G06F 11/0781: {Error filtering or prioritizing based on a policy defined by the user or on a policy defined by a hardware/software module, e.g. according to a severity level} - G06F 11/3006: {where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems (multiprogramming arrangements G06F9/46; allocation of resources G06F9/50)} - G06F 11/3409: {for performance assessment} - G06F 16/2365: {Ensuring data consistency and integrity} - G06N 20/00: Machine learning - G06F 11/076: {by exceeding a count or rate limit, e.g. word- or bit count limit} - G06F 11/079: {Root cause analysis, i.e. error or fault diagnosis (in a hardware test environment G06F11/22; in a software test environment G06F11/36)} - G06F 11/0793: {Remedial or corrective actions (recovery from an exception in an instruction pipeline G06F9/3861; by retry G06F11/1402; for recovering from a failure of a protocol instance or entity H04L69/40)} - G06F 11/3058: {Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations (thermal management in cooling arrangements of a computing system G06F1/206)} - G06F 11/3447: {Performance evaluation by modeling} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract A computerized method is disclosed for automated handling of data ingestion anomalies. The method features operations of detecting a data ingestion anomaly and determining a cause for the data ingestion anomaly. The causal determination may be conducted by at least (i) determining features of an anomalous data ingestion volume, (ii) training a second data model, after a first data model being used to detect the data ingestion anomaly, with data sets consistent with the determined features, (iii) applying the second data model to predict whether a data ingestion sub-volume is anomalous, (iv) obtaining system state information during ingestion of the anomalous data ingestion sub-volume, and (v) determining the cause of the anomalous data ingestion volume based on the system state information. ## Claims ### Claim 1 A computerized method comprising: detecting a data ingestion anomaly through deployment of a first machine learning model taking a data ingestion volume as input, wherein detection of the data ingestion anomaly indicates that the data ingestion volume is an anomalous data ingestion volume; determining a cause for the data ingestion anomaly by at least: (i) determining features of the anomalous data ingestion volume, (ii) training a second machine learning model with historical data sets consistent with the determined features of the anomalous data ingestion volume, (iii) deploying a second machine learning model taking a data ingestion sub-volume as input, wherein the second machine learning model is configured to predict whether the data ingestion sub-volume is anomalous, wherein the data ingestion sub-volume is a portion of the anomalous data ingestion volume, (iv) obtaining system state information during ingestion of the anomalous data ingestion sub-volume, and (v) determining the cause of the anomalous data ingestion volume based on the system state information; and generating a graphical user interface (GUI) that illustrates a graphical representation of the data ingestion volume over a prescribed time period and a visual representation of an error threshold relative to a predicted data ingestion volume over the prescribed time period, wherein the data ingestion anomaly is displayed as a visually distinct visual element from the graphical representation of the data ingestion volume and is displayed outside of the visual representation of the error threshold, and wherein the GUI displays a remedial recommendation for resolving the cause of the anomalous data ingestion value including an indication to fix or replace a component that is inoperable. ### Claim 2 (depends on 1) The computerized method of claim 1, wherein the features of the anomalous data ingestion volume comprise a first feature including information identifying a day of a week of receipt of the anomalous data ingestion volume, a second feature including information identifying a time of the day of receipt of the anomalous data ingestion volume, and a third feature directed to whether the day of the week corresponds to a holiday. ### Claim 3 (depends on 1) The computerized method of claim 1, wherein the system state information includes information identify a processor overheating condition or a memory storage overflow condition. ### Claim 4 (depends on 1) The computerized method of claim 1 further comprising: providing remediation recommendations to an administrator with suggested fixes to mitigate or avoid further data ingestion anomalies after the data ingestion anomaly has occurred. ### Claim 5 (depends on 1) The computerized method of claim 1, wherein the detecting of the data ingestion anomaly comprises: training the first machine learning model; conducting, using the first machine learning model, a predictive analysis of a data ingestion volume to produce a predicted data ingestion volume; conducting a correlative analysis between the predicted data ingestion volume and the data ingestion volume during a prescribed time period to produce a prediction error; and based on the prediction error exceeding at least a first threshold, detecting that the data ingestion volume constitutes the anomalous data ingestion volume. ### Claim 6 (depends on 5) The computerized method of claim 5 further comprising: generating a notification, when the data ingestion volume constitutes the anomalous data ingestion volume, based at least on a determination whether the prediction error is equal to or greater than the first threshold or a second threshold greater than the first threshold. ### Claim 7 (depends on 6) The computerized method of claim 6, wherein the second threshold constitutes a prediction error range equal to or exceeding at least three standard deviations from a mean prediction error value determined over a prescribed duration prior to receipt of the data ingestion volume. ### Claim 8 (depends on 6) The computerized method of claim 6, wherein the generating of the notification comprises generating a warning message when the prediction error falls between the first threshold and the second threshold, the warning message operating as a type of dashboard alert signifying the data ingestion anomaly of a first severity has been detected. ### Claim 9 (depends on 8) The computerized method of claim 8, wherein the generating of the notification further comprises generating an alert message when the prediction error is equal to or exceeds the second threshold, the alert message operating at least as a type of dashboard alert signifying the data ingestion anomaly of a second severity that is more severe than the data ingestion anomaly of the first severity. ### Claim 10 (depends on 1) The computerized method of claim 1, wherein the component that is inoperable is either a component of an upstream data pipeline or a data source. ### Claim 11 A computing device, comprising: one or more streaming data processors; and a non-transitory storage medium having stored thereon an intake system being software including a plurality of components forming a data ingestion analytic subsystem, the data ingestion analytic subsystem comprises: a model training component configured to train (i) a first machine learning model of one or more machine learning models based on a first volume of data and (ii) a second machine learning model after determining a data ingestion volume is anomalous, and wherein the second machine learning model is trained on historical data sets consistent with features determined from an anomalous data ingestion volume as determined by deployment of the first machine learning model, a prediction component configured to conduct, using the second machine learning model, a predictive analysis of a sub-volume of the anomalous data ingestion volume to produce a predicted data ingestion sub-volume, wherein sub-volume is a portion of the anomalous data ingestion volume, an anomaly detection component configured to conduct a correlative analysis between the predicted data ingestion sub-volume and an actual data ingestion sub-volume to produce a prediction error, and an anomaly classification component configured to determine a degree of anomalousness of the prediction error, and a notification generation component configured to (i) generate a graphical user interface (GUI) providing a notification based on the degree of anomalousness associated with the prediction error and (ii) determine a cause of the anomalous data ingestion volume by at least (a) obtaining system state information during ingestion of the anomalous data ingestion sub-volume and (b) determining the cause of the anomalous data ingestion volume based on the system state information, wherein the GUI illustrates a graphical representation of the data ingestion volume over a prescribed time period and a visual representation of an error threshold relative to a predicted data ingestion volume over the prescribed time period, wherein a data ingestion anomaly is displayed as a visually distinct visual element from the graphical representation of the data ingestion volume and is displayed outside of the visual representation of the error threshold, and wherein the GUI displays a remedial recommendation for resolving the cause of the anomalous data ingestion value including an indication to fix or replace a component that is inoperable. ### Claim 12 (depends on 11) The computing device of claim 11, wherein the anomalous data ingestion volume comprise a first feature including information identifying a day of a week of receipt of the anomalous data ingestion volume, a second feature including information identifying a time of the day of receipt of the anomalous data ingestion volume, and a third feature directed to whether the day of the week corresponds to a holiday. ### Claim 13 (depends on 11) The computing device of claim 11, wherein the system state information includes information identify a processor overheating condition or a memory storage overflow condition. ### Claim 14 (depends on 11) The computing device of claim 11, wherein the remedial recommendation provides suggested fixes to mitigate or avoid further data ingestion anomalies after the data ingestion anomaly has occurred. ### Claim 15 (depends on 11) The computing device of claim 11, wherein the prediction component, the anomaly detection component and the anomaly classification component are configured to collectively operate to detect the data ingestion anomaly by at least: training the first machine learning model; conducting, using the first [data] machine learning model, a predictive analysis of a data ingestion volume to produce a predicted data ingestion volume; conducting a correlative analysis between the predicted data ingestion volume and the data ingestion volume during a prescribed time period to produce a prediction error; and based on the prediction error exceeding at least a first threshold, detecting that the data ingestion volume constitutes the anomalous data ingestion volume. ### Claim 16 (depends on 15) The computing device of claim 15, wherein the notification generation component is configured to generate the notification based at least on a determination that the prediction error is equal to or greater than a second threshold being greater than the first threshold. ### Claim 17 (depends on 16) The computing device of claim 16, wherein the second threshold constitutes a prediction error range equal to or exceeding at least three standard deviations from a mean prediction error value determined over a prescribed duration prior to receipt of the data ingestion volume. ### Claim 18 (depends on 16) The computing device of claim 16, wherein the notification generation component is configured to generate the notification by at least generating a warning message when the prediction error falls between the first threshold and the second threshold, the warning message operating as a type of dashboard alert signifying the data ingestion anomaly of a first severity has been detected. ### Claim 19 (depends on 18) The computing device of claim 18, wherein the notification generation component is configured to generate the notification by at least generating an alert message when the prediction error is equal to or exceeds the second threshold, the alert message operating at least as a type of dashboard alert signifying the data ingestion anomaly of a second severity that is more severe than the data ingestion anomaly of the first severity. ### Claim 20 (depends on 11) The computing device of claim 11, wherein the component that is inoperable is either a component of an upstream data pipeline or a data source. ### Claim 21 A non-transitory storage medium having stored thereon software that, when executed, is configured to perform operations comprising: detecting a data ingestion anomaly through deployment of a first machine learning model taking a data ingestion volume as input, wherein detection of the data ingestion anomaly indicates that the data ingestion volume is an anomalous data ingestion volume; determining features of the anomalous data ingestion volume; training a second machine learning model with historical data sets consistent with the determined features of the anomalous data ingestion volume; deploying a second machine learning model taking a data ingestion sub-volume as input, wherein the second machine learning model is configured to predict whether the data ingestion sub-volume is anomalous, wherein the data ingestion sub-volume is a portion of the anomalous data ingestion volume; obtaining system state information during ingestion of the anomalous data ingestion sub-volume; determining the cause of the anomalous data ingestion volume based on the system state information; and generating a graphical user interface (GUI) that illustrates a graphical representation of the data ingestion volume over a prescribed time period and a visual representation of an error threshold relative to a predicted data ingestion volume over the prescribed time period, wherein the data ingestion anomaly is displayed as a visually distinct visual element from the graphical representation of the data ingestion volume and is displayed outside of the visual representation of the error threshold, and wherein the GUI displays a remedial recommendation for resolving the cause of the anomalous data ingestion value including an indication to fix or replace a component that is inoperable. ### Claim 22 (depends on 21) The non-transitory storage medium of claim 21, wherein the features of the anomalous data ingestion volume comprise a first feature including information identifying a day of a week of receipt of the anomalous data ingestion volume, a second feature including information identifying a time of the day of receipt of the anomalous data ingestion volume, and a third feature directed to whether the day of the week corresponds to a holiday. ### Claim 23 (depends on 21) The non-transitory storage medium of claim 21, wherein the system state information includes information identify a processor overheating condition or a memory storage overflow condition. ### Claim 24 (depends on 21) The non-transitory storage medium of claim 21, wherein the remedial recommendation provides suggested fixes to mitigate or avoid further data ingestion anomalies after the data ingestion anomaly has occurred. ### Claim 25 (depends on 21) The non-transitory storage medium of claim 21, wherein the software is configured to detect the data ingestion anomaly by at least performing operations comprising: training the first machine learning model; conducting, using the first machine learning model, a predictive analysis of a data ingestion volume to produce a predicted data ingestion volume; conducting a correlative analysis between the predicted data ingestion volume and the data ingestion volume during a prescribed time period to produce a prediction error; and based on the prediction error exceeding at least a first threshold, detecting that the data ingestion volume constitutes the anomalous data ingestion volume. ### Claim 26 (depends on 25) The non-transitory storage medium of claim 25, wherein the software is further configured to perform operations, comprising: generating a notification, when the data ingestion volume constitutes the anomalous data ingestion volume, based at least on a determination whether the prediction error is equal to or greater than the first threshold or a second threshold greater than the first threshold. ### Claim 27 (depends on 26) The non-transitory storage medium of claim 26, wherein the second threshold constitutes a prediction error range equal to or exceeding at least three standard deviations from a mean prediction error value determined over a prescribed duration prior to receipt of the data ingestion volume. ### Claim 28 (depends on 26) The non-transitory storage medium of claim 26, wherein the software being configured to generate the notification by at least generating a warning message when the prediction error falls between the first threshold and the second threshold, the warning message operating as a type of dashboard alert signifying the data ingestion anomaly of a first severity has been detected. ### Claim 29 (depends on 28) The non-transitory storage medium of claim 28, wherein the software being configured to generate the notification by at least generating of the notification further comprises generating an alert message when the prediction error is equal to or exceeds the second threshold, the alert message operating at least as a type of dashboard alert signifying the data ingestion anomaly of a second severity that is more severe than the data ingestion anomaly of the first severity. ### Claim 30 (depends on 21) The non-transitory storage medium of claim 21, wherein the component that is inoperable is either a component of an upstream data pipeline or a data source.