Back to results
US 11,960,471 B2Granted
Using lineage to infer data quality issues
AssigneeMicrosoft Technology Licensing, LLC
InventorJeffrey Michael Derstadt
Life of this patent
Estimating the term…
Abstract
Identifying data quality along a data flow. A method includes identifying quality metadata for two or more datasets. The quality metadata defines one or more of quality of a data source, accuracy of a dataset, completeness of a dataset, freshness of a dataset, or relevance of a dataset. At least some of the metadata is based on results of operations along a data flow. Based on the metadata, the method includes creating one or more quality indexes for the datasets. The one or more quality indexes include a characterization of quality of two or more datasets.
Claims20
A method comprising:
- performing an operation to generate a first dataset at a first data source from a second dataset at a second data source;
- identifying quality metadata for at least one of the second dataset or the second data source;
- inferring, from the quality metadata, an inferred quality of metadata of the first dataset such that the inferred quality of the first dataset can be inferred without direct access to data in the first dataset;
- determining an actual quality of the data in the first dataset;
- comparing the inferred quality of the data and the actual quality of the first dataset to identify differences in the inferred quality of the first dataset and the actual quality of the first dataset;
- and
- based on the differences between the inferred quality of the first dataset compared with the actual quality of the first dataset, displaying, on a user interface, an indication of an effectiveness of at least one operation in a data flow.
4 more elements
A computer program product comprising a computer storage device containing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to perform a computer-implemented method comprising:
- performing an operation to generate a first dataset at a first data source from a second dataset at a second data source;
- identifying quality metadata for at least one of the second dataset, or the second data source,
- inferring, from the quality metadata, an inferred quality of metadata of the first dataset such that the inferred quality of the first dataset can be inferred without direct access to data in the first dataset;
- determining an actual quality of the data in the first dataset;
- comparing the inferred quality of the first dataset to the actual quality of the first dataset to identify differences in the inferred quality of the first dataset and the actual quality of the first dataset;
- and
- based on the differences between the inferred quality of the first dataset compared with the actual quality of the first dataset, displaying on a user interface an indication of an effectiveness of at least one operation in a data flow.
4 more elements
A computing system comprising:
- one or more processors;
- a computer program product comprising a computer storage device containing computer-executable instructions which, when executed by one or more processors, cause the one or more processors to implement a computer-implemented method comprising:
- performing an operation to generate a first dataset at a first data source for a data flow from a second dataset at a second data source in the data flow;
- identifying quality metadata for at least one of the second dataset, or the second data source, the quality metadata defining one or more of the following:
- quality of the second data source in the data flow,
- correctness of the second dataset by defining a measure of how well the second dataset accurately reflects what the second dataset purports to represent without errors,
- completeness of the second dataset by defining a measure of inclusion of predetermined elements in the second dataset,
- freshness of the second dataset by defining a measure of how recently data in the second dataset was produced with respect to how often data in the second dataset is expected to change, or
- relevance of the second dataset by defining a measure of at least one of popularity of the second dataset or contextual relevance of the second dataset to a system in which the second dataset is implemented;
- inferring, from the quality metadata, an inferred quality of metadata of the first dataset such that the inferred quality of the first dataset can be inferred without direct access to data in the first dataset;
- determining an actual quality of the data in the first dataset;
- comparing the inferred quality of the first dataset to the actual quality of the first dataset to identify differences in the inferred quality of the first dataset and the actual quality of the first dataset;
- and
- based on the differences between the inferred quality of the first dataset compared with the actual quality of the first dataset, displaying on a user interface an indication of an effectiveness of at least one operation in the data flow.
11 more elements
Dependencies are inferred from the claim text.