US 2023/0409587 A1 · DATA RECOMMENDER USING LINEAGE TO PROPAGATE VALUE INDICATORS

Abstract

Systems and methods provide a system that gathers information about data as it progresses through data processing pipelines of data analysis projects. The data analytics system derives value indicators and implicit metadata from the data processing pipelines. For example, the data analytics system may derive value indicators and implicit metadata from data-related products themselves, semantic analysis of the code/processing steps used to process the data-related products, the structure of data processing pipelines, and human behavior related to production and usage of data-related products. Once a new data analysis project is initiated, the data analytics system gathers parameters and characteristics about the new data analysis project and references the value indicators and implicit metadata to recommend useful processing steps, datasets, and/or other data-related products for the new data analysis project.

Claims20

Dependent claims

A system comprising:

one or more processing resources;
and
a non-transitory computer-readable medium, coupled to the one or more processing resources, having stored therein instructions that when executed by the one or more processing resources cause the system to:
extract and characterize metadata from data processing pipelines, wherein extracting and characterizing metadata from the data processing pipelines comprises generating lineage representations for the data processing pipelines;
catalog the characterized metadata;
propagate value indicators through the lineage representations for the data processing pipelines;
based on the propagation of the value indicators, identify a subset of data-related products of the data processing pipelines having highest relative value;
based on the characterized metadata, identify a high value-influencing metadata characteristic that arises with statistically significant frequency among the subset of data-related products having the highest relative value;
and
recommend one or more data-related products having the high-value influencing metadata characteristic for a target data analysis project.

7 more elements

A method comprising:

extracting and characterizing metadata from data processing pipelines, wherein extracting and characterizing the metadata from the data processing pipelines comprises generating lineage representations for the data processing pipelines;
cataloging the extracted metadata from the data processing pipelines;
propagating value indicators through the lineage representations for the data processing pipelines;
based on the propagation of the value indicators, identifying a subset of data-related products of the data processing pipelines having highest relative value;
based on the characterized metadata, identifying a high value-influencing metadata characteristic that arises with statistically significant frequency among the subset of data-related products having the highest relative value;
and
recommend one or more data-related products having the high-value influencing metadata characteristic for a target data analysis project.

4 more elements

Dependencies are inferred from the claim text.

Data recommender using lineage to propagate value indicators

Life of this patent

Data recommender using lineage to propagate value indicators

Life of this patent

Abstract

Claims20