Training and implementing a data quality verification model to validate recurring data pipelines

A system, comprising:

• at least one processor;
• memory in electronic communication with the at least one processor;
• instructions stored in the memory, the instructions being executable by the at least one processor to:
• identify a false positive rate (FPR) threshold associated with a rate of falsely predicting data quality (DQ) errors within a plurality of recurring data pipelines;
• train a pipeline error detection model to output a prediction indicating whether a given execution of a recurring data pipeline has a data quality (DQ) error, wherein training the pipeline error detection model includes:
• constructing a collection of DQ constraints for the recurring data pipeline, the collection of DQ constraints indicating a plurality of statistical measures and associated ranges;
• identifying a set of training data pipelines having similar features as the plurality of recurring data pipelines;
• and
• iteratively applying DQ variants associated with respective DQ constraints from the collection of DQ constraints to the set of training data pipelines, the DQ variants comprising changes to one or more values of the set of training data pipelines;
• and
• determining a subset of DQ constraints from the collection of DQ constraints based on the one or more DQ constraints providing a maximized rate of recall while staying within a budget of the FPR threshold;
• and
• compiling the subset of DQ constraints within the trained pipeline error detection model such that when applied to the given execution of the recurring data pipeline, the pipeline-error detection model is configured to generate an output including a prediction of whether a DQ error exists within the given execution of the recurring data pipeline.

A non-transitory computer readable media storing instructions thereon that, when executed by at least one processor, causes a computing device to:

• identify a false positive rate (FPR) threshold associated with a rate of falsely predicting data quality (DQ) errors within a plurality of recurring data pipelines;
• train a pipeline error detection model to output a prediction indicating whether a given execution of a recurring data pipeline has a data quality (DQ) error, wherein training the pipeline error detection model includes:
• constructing a collection of DQ constraints for the recurring data pipeline, the collection of DQ constraints indicating a plurality of statistical measures and associated ranges;
• identifying a set of training data pipelines having similar features as the plurality of recurring data pipelines;
• and
• iteratively applying DQ variants associated with respective DQ constraints from the collection of DQ constraints to the set of training data pipelines, the DQ variants comprising changes to one or more values of the set of training data pipelines;
• determining a subset of DQ constraints from the collection of DQ constraints based on the one or more DQ constraints providing a maximized rate of recall while staying within a budget of the FPR threshold;
• and
• compiling the subset of DQ constraints within the trained pipeline error detection model such that when applied to the given execution of the recurring data pipeline, the pipeline-error detection model is configured to generate an output including a prediction of whether a DQ error exists within the given execution of the recurring data pipeline.