US 12,216,625 B2 · Elastic data sampling in a data pipeline

Abstract

Various embodiments comprise systems and methods to sample data outputs of a data pipeline. In some examples, data monitoring circuitry monitors the data pipeline wherein the data pipeline receives an input data set, processes the input data set, and responsively generates and transfers an output data set. The data monitoring circuitry ingests the output data set, determines an amount of available computing resources, and selects an amount of the values from the output data set based on the amount of available computing resources. The data monitoring circuitry generates a quality score for the selected values based on a data quality, generates a confidence score based on the amount of the selected values, and reports the quality score and the confidence score.

Claims20

Dependent claims

A data pipeline monitoring system configured to sample data outputs of a data pipeline, the data pipeline monitoring system comprising:

data monitoring circuitry configured to:
monitor the data pipeline wherein the data pipeline receives an input data set, processes the input data set, and responsively generates and transfers an output data set;
ingest the output data set;
query a cloud computing service to determine an amount of available computing resources to process the output data set;
host a data structure that correlates amounts of available computing resources to output data set sample sizes;
input the amount of available computing resources into the data structure to determine a sample size for the output data set;
sample data values from the output data set based on the sample size;
utilize the available computing resources to process the sampled data values to determine a data quality score for the output data set;
host another data structure that correlates the output data set sample sizes to confidence scores for data quality scores;
input the sample size into the other data structure to determine a confidence score for the data quality score;
and
report the quality score and the confidence score.

9 more elements

A method of operating a data pipeline monitoring system to sample data outputs of a data pipeline, the method comprising:

by data monitoring circuitry in the data pipeline monitoring system:
monitoring the data pipeline wherein the data pipeline receives an input data set, processes the input data set, and responsively generates and transfers an output data set;
ingesting the output data set;
querying a cloud computing service to determine an amount of available computing resources to process the output data set;
hosting a data structure that correlates amounts of available computing resources to output data set sample sizes;
inputting the amount of available computing resources into the data structure to determine a sample size for the output data set;
sampling data values from the output data set based on the sample size;
utilizing the available computing resources to process the sampled data values to determine a data quality score for the output data set;
hosting another data structure that correlates the output data set sample sizes to confidence scores for data quality scores;
inputting the sample size into the other data structure to determine a confidence score for the data quality score;
and
reporting the quality score and the confidence score.

9 more elements

A non-transitory computer-readable medium storing instructions to sample data outputs of a data pipeline, wherein the instructions, in response to execution by one or more processors, cause the one or more processors to drive a system to perform operations comprising:

monitoring the data pipeline wherein the data pipeline receives an input data set, processes the input data set, and responsively generates and transfers an output data set;
ingesting the output data set;
querying a cloud computing service to determine an amount of available computing resources to process the output data set;
hosting a data structure that correlates amounts of available computing resources to output data set sample sizes;
inputting the amount of available computing resources into the data structure to determine a sample size for the output data set;
sampling data values from the output data set based on the sample size;
utilizing the available computing resources to process the sampled data values to determine a data quality score for the output data set;
hosting another data structure that correlates the output data set sample sizes to confidence scores for data quality scores;
inputting the sample size into the other data structure to determine a confidence score for the data quality score;
and
reporting the quality score and the confidence score.

8 more elements

Dependencies are inferred from the claim text.

Elastic data sampling in a data pipeline

Life of this patent

Elastic data sampling in a data pipeline

Life of this patent

Abstract

Claims20