Column lineage and metadata propagation

#This disclosure relates to the field of data aggregation and analysis systems, and in particular to column metadata generation for a dataset.

#Modern data centers often comprise thousands of hosts that operate collectively to service requests from even larger numbers of remote clients. During operation, components of these data centers can produce significant volumes of machine-generated data. A computerized data pipeline is a useful mechanism for processing those large amounts of data. A typical data pipeline is an ad-hoc collection of computer software scripts and programs for processing data extracted from “data sources” and for providing the processed data to “data sinks.” As an example, a data pipeline for a large insurance company that has recently acquired a number of smaller insurance companies may extract policy and claim data from the individual database systems of the smaller insurance companies, transform and validate the insurance data in some way, and provide validated and transformed data to various analytical platforms for assessing risk management, compliance with regulations, fraud, etc.

#Between the data sources and the data sinks, a data pipeline system is typically provided as a software platform to automate the movement and transformation of data from the data sources to the data sinks. In essence, the data pipeline system shields the data sinks from having to interface with the data sources or even being configured to process data in the particular formats provided by the data sources. Typically, data from the data sources received by the data sinks is processed by the data pipeline system in some way. For example, a data sink may receive data from the data pipeline system that is a combination (e.g., a join) of data of from multiple data sources, all without the data sink being configured to process the individual constituent data formats.

#The present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments thereof, which, however, should not be taken to limit the present disclosure to the specific embodiments, but are for explanation and understanding only.

#FIG. 1 is a block diagram illustrating a network environment in which a metadata management system may operate, in accordance with some embodiments.

#FIG. 2 is a block diagram illustrating a metadata management system, in accordance with some embodiments.

#The following description sets forth numerous specific details such as examples of specific systems, components, methods, and so forth, in order to provide a good understanding of several embodiments of the present disclosure. It will be apparent to one skilled in the art, however, that at least some embodiments of the present disclosure may be practiced without these specific details. In other instances, well-known components or methods are not described in detail or are presented in simple block diagram format in order to avoid unnecessarily obscuring the present disclosure. Thus, the specific details set forth are merely exemplary. Particular embodiments may vary from these exemplary details and still be contemplated to be within the scope of the present disclosure.

#Aspects of the present disclosure are directed to metadata generation for columns of a dataset. The dataset may be used or created as part of a data pipeline. A data pipeline may refer to an ordered set of logic (e.g., a collection of computer software scripts or programs) that performs a multi-step transformation of data obtained from data sources to produce one or more output datasets. Each data transformation step applies transformation code to one or more source datasets (i.e. collections of enterprise data) to produce one or more target datasets. For example, the transformation code can be software code, such as a script, that defines a set of instructions to transform source columns of one or more source datasets into target columns of one or more target datasets. In a data pipeline, the source dataset can result in hundreds if not thousands of derived (target) datasets. For instance, in a first data transformation step of the data pipeline, first transformation code can be applied to the source dataset which represents the starting point of the data pipeline, transforming the source dataset into one or more target datasets. The target dataset(s) can be further transformed using second transformation code to provide one or more other target datasets as input to the next data transformation step using different transformation code, and so forth.

#Columns in a dataset can be associated with column metadata that provides information about a particular column. Column metadata can include user comment metadata (e.g., user comments for a respective column), and column level access control metadata (e.g., policy to restrict access to a respective column or the data therein to persons or operations having adequate authority). In addition, column metadata can include column lineage metadata, which describes a column's provenance (also referred to as “lineage” herein), such that the column lineage metadata identifies the origins and history of a column through the column's life cycle (e.g., from the source column(s) in the first source dataset to the target column in a target dataset from a particular data transformation step in the data pipeline). For instance, column lineage metadata can describe each transformation step of column through its life cycle and at each transformation step can describe the source column(s) in one or more source datasets used to generate one or more columns in one or more target datasets, identify the history (e.g., earlier iterations or transformations) of the source column(s), identify how the source column(s) were transformed (e.g., renamed, database operations, function, etc.), and identify the transformation code used at the transformation step. Provenance of a dataset, and in particular column provenance of columns of a dataset, can help users or systems determine whether a given dataset is trustworthy. An error or mistake in a dataset can propagate through the data pipeline if left uncorrected. Such an error or mistake can cause many problems including, for example, inaccurate data, failure of downstream processes that rely on the dataset, and so forth. Column lineage metadata provides granularity with respect to a column's history, which can be invaluable for identifying and correcting propagated errors in datasets.