Automated data observability system

A method comprising:

• periodically obtaining metadata relating to respective nodes of a plurality of nodes;
• automatically generating data quality metrics for respective nodes of the plurality of nodes based on the metadata;
• obtaining predicted values for the data quality metrics including by:
• executing a plurality of candidate machine learning models over time series data including measured values corresponding to respective ones of the data quality metrics to generate candidate predicted values for respective ones of the data quality metrics,
• selecting a selected machine learning model from the plurality of candidate machine learning models for respective ones of the data quality metrics based on a comparison between at least some of the measured values and at least some of the predicted values, and
• obtaining the predicted values based on the selected machine learning model for respective ones of the data quality metrics;
• determining a first anomaly relating to a first one of the data quality metrics for a first node of the plurality of nodes including by comparing a first measured value at a first time relating to the first one of the data quality metrics and a first predicted value at the first time relating to the first one of the data quality metrics;
• generating a data pipeline graph corresponding to the at least some of the plurality of nodes including the first node by parsing logs including information relating to usage of at least some of the plurality of nodes to identify relationships between at least some of the plurality of nodes, including creating a directional edge from a first table to a second table based on an identification of a pattern of inserts into the second table based on data selected from the first table, wherein the data pipeline graph includes information indicative of an importance of the directional edge based on the pattern of inserts;
• estimating a root cause relating to the first anomaly by traversing the data pipeline graph from the first node;
• determining a priority of the estimated root cause based on information indicative of the importance of directional edges between a second node associated with the root cause and the first node, wherein the priority is further determined based on an identity of users accessing the first node, second node, or nodes between the first node and second node and a frequency of users accessing the first node, second node, or nodes between the first node and second node;
• and
• transmitting information relating to the relating to the estimated root cause to a client device for display on the client device based on the priority,
• wherein the nodes of the plurality of nodes are tables in a data warehouse system, the logs are query logs including queries executed by the data warehouse system.

A non-transitory computer readable medium storing instructions that when executed by a processor cause the processor to:

• periodically obtain metadata relating to respective nodes of a plurality of nodes;
• automatically generate data quality metrics for respective nodes of the plurality of nodes based on the metadata;
• obtain predicted values for the data quality metrics including by:
• executing a plurality of candidate machine learning models over time series data including measured values corresponding to respective ones of the data quality metrics to generate candidate predicted values for respective ones of the data quality metrics,
• selecting a selected machine learning model from the plurality of candidate machine learning models for respective ones of the data quality metrics based on a comparison between at least some of the measured values and at least some of the predicted values, and
• obtaining the predicted values based on the selected machine learning model for respective ones of the data quality metrics;
• determine a first anomaly relating to a first one of the data quality metrics for a first node of the plurality of nodes including by comparing a first measured value at a first time relating to the first one of the data quality metrics and a first predicted value at the first time relating to the first one of the data quality metrics;
• generate a data pipeline graph corresponding to at least some of the plurality of nodes by parsing logs including information relating to usage of at least some of the plurality of nodes to identify relationships between at least some of the plurality of nodes, including creating a directional edge from a first table to a second table based on an identification of a pattern of inserts into the second table based on data selected from the first table, wherein the data pipeline graph includes information indicative of an importance of the directional edge based on the pattern of inserts;
• estimate a root cause relating to the first anomaly by traversing the data pipeline graph from the first node;
• determine a priority of the estimated root cause based on information indicative of the importance of directional edges between a second node associated with the root cause and the first node, wherein the priority is further determined based on an identity of users accessing the first node, second node, or nodes between the first node and second node and a frequency of users accessing the first node, second node, or nodes between the first node and second node;
• and
• transmit, to a client device, information relating to the estimated root cause for display by the client device based on the priority,
• wherein the nodes of the plurality of nodes are tables in a data warehouse system, the logs are query logs including queries executed by the data warehouse system.