Self-analyzing data processing job to determine data quality issues
AssigneeInternational Business Machines Corporation
InventorsJeff J. Li, Yong Li
Life of this patent
Estimating the term…
Abstract
Techniques are disclosed to determine data quality issues in data processing jobs. The data processing job is received, the data processing job specifying one or more processing steps designed based on one or more data schemas and further specifies one or more desired quality metrics to measure at the one or more processing steps. One or more state machines are provided, that are generated based on the quality metrics and on the data schemas. Input data to the data process job are processed using the one or more state machines, in order to generate output data and a set of data quality records characterizing a set of data quality issues identified during the execution of the data processing job.
Claims23
A computer-implemented method to determine data quality issues in data processing jobs, the method comprising:
- receiving a data processing job specifying one or more processing steps designed based on one or more data schemas and further specifying one or more desired quality metrics to measure at the one or more processing steps;
- providing one or more state machines generated based on the quality metrics and on the data schemas;
- processing input data for the data processing job by operation of one or more computer processors and using the one or more state machines, in order to generate output data for the data processing job and a set of data quality records characterizing a set of data quality issues identified during execution of the data processing job;
- and
- outputting the generated set of data quality records.
A computer program product to determine data quality issues in data processing jobs, the computer program product comprising:
- a computer-readable storage medium having program code embodied therewith, the program code executable by one or more computer processors to:
- receive a data processing job specifying one or more processing steps designed based on one or more data schemas and further specifying one or more desired quality metrics to measure at the one or more processing steps;
- provide one or more state machines generated based on the quality metrics and on the data schemas;
- process input data for the data processing job using the one or more state machines, in order to generate output data for the data processing job and a set of data quality records characterizing a set of data quality issues identified during execution of the data processing job;
- and
- output the generated set of data quality records.
A system to determine data quality issues in data processing jobs, the system comprising:
- one or more computer processors;
- a memory containing a program which, when executed by the one or more computer processors, is configured to perform an operation comprising:
- receiving a data processing job specifying one or more processing steps designed based on one or more data schemas and further specifying one or more desired quality metrics to measure at the one or more processing steps;
- providing one or more state machines generated based on the quality metrics and on the data schemas;
- processing input data for the data processing job using the one or more state machines, in order to generate output data for the data processing job and a set of data quality records characterizing a set of data quality issues identified during execution of the data processing job;
- and
- outputting the generated set of data quality records.
Dependencies are inferred from the claim text.