Using lineage to infer data quality issues
AssigneeMicrosoft Corporation
InventorJeffrey Michael Derstadt
Life of this patent
Estimating the term…
Abstract
Identifying data quality along a data flow. A method includes identifying quality metadata for two or more datasets. The quality metadata defines one or more of quality of a data source, accuracy of a dataset, completeness of a dataset, freshness of a dataset, or relevance of a dataset. At least some of the metadata is based on results of operations along a data flow. Based on the metadata, the method includes creating one or more quality indexes for the datasets. The one or more quality indexes include a characterization of quality of two or more datasets.
Claims21
-20. (canceled)
A computer-implemented method of identifying data quality for data that transferred along a data flow, the computer-implemented method being performed by one or more processors when executing computer-executable instructions, and comprising:
- receiving two or more datasets from one or more sources of data;
- identifying quality metadata for the two or more datasets, the quality metadata defining one or more of the following:
- quality of a data source,
- accuracy of a dataset,
- completeness of a dataset,
- freshness of a dataset, or
- relevance of a dataset;
- wherein at least some of the identified quality metadata is based on results of operations performed on at least one dataset as the at least one dataset is transferred along a data flow;
- and
- based on the identified quality metadata, creating at least one quality index for at least one dataset, the at one quality index being based on assessed quality of the two or more datasets.
The computer-implemented of claim 20, further comprising:
- performing an operation on two or more of the sub-datasets to create an output dataset;
- and
- identifying a quality for the output dataset based on the quality of the two or more sub-datasets used to create the output dataset.
A computer program product comprising a computer storage device containing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to perform a computer-implemented method of identifying data quality for data that transferred along a data flow, and wherein the computer-implemented method comprises:
- receiving two or more datasets from one or more sources of data;
- identifying quality metadata for the two or more datasets, the quality metadata defining one or more of the following:
- quality of a data source,
- accuracy of a dataset,
- completeness of a dataset,
- freshness of a dataset, or
- relevance of a dataset;
- wherein at least some of the identified quality metadata is based on results of operations performed on at least one dataset as the at least one dataset is transferred along a data flow;
- and
- based on the identified quality metadata, creating at least one quality index for at least one dataset, the at one quality index being based on assessed quality of the two or more datasets.
A computing system comprising:
- one or more processors;
- a computer program product comprising a computer storage device containing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to implement a computer-implemented method of identifying data quality for data that transferred along a data flow, and wherein the computer-implemented method comprises:
- receiving two or more datasets from one or more sources of data;
- identifying quality metadata for the two or more datasets, the quality metadata defining one or more of the following:
- quality of a data source,
- accuracy of a dataset,
- completeness of a dataset,
- freshness of a dataset, or
- relevance of a dataset;
- wherein at least some of the identified quality metadata is based on results of operations performed on at least one dataset as the at least one dataset is transferred along a data flow;
- and
- based on the identified quality metadata, creating at least one quality index for at least one dataset, the at one quality index being based on assessed quality of the two or more datasets.
Dependencies are inferred from the claim text.