Back to results
US 12,591,556 B1Granted
Data lineage tracking
AssigneeDatabricks, Inc.
InventorsTao Feng, Menglei Sun, Zhuoying Wang
Life of this patent
Estimating the term…
Abstract
The present application discloses a method, system, and computer system for managing lineage data for data entities. The method includes generating lineage data, wherein generating the lineage data, and storing and indexing, in a data structure, the lineage data in association with the selected data entity. The generating the lineage data includes selecting a selected data entity, obtaining a query tree that was used to generate the selected data entity, and determining lineage data for the selected data entity based at least in part on the query tree.
Claims19
A system, comprising:
- a processor;
- and
- a memory storing instructions that, when executed by the processor, cause the processor to perform actions comprising:
- receiving from a user of a client device, a request to execute a query on one or more input data entities;
- parsing the query to generate a query tree for the query where the query tree includes one or more nodes that correspond to operations defined by the query, wherein the one or more nodes of the query tree includes a root node and one or more leaf nodes;
- executing on a computing cluster, the query based on the query tree to generate at least one generated data entity, wherein the generated data entity is stored in association with a data entity identifier;
- generating a lineage data for the generated data entity by traversing the generated query tree starting from the root node to the one or more leaf nodes of the query tree, wherein the lineage data includes at least one or more dependencies for a column of the generated data entity relative to one or more columns of the input data entities;
- storing the generated lineage data in a database in association with a lineage identifier that is mapped to the data entity identifier;
- configuring an application programming interface (API) to expose the generated lineage data in the database;
- receiving a second request for lineage data associated with the generated data entity via the API;
- retrieving the generated lineage data associated with the generated data entity from the database, wherein retrieving the generated lineage data further comprises:
- obtaining the data entity identifier for the generated data entity,
- obtaining the lineage identifier that is mapped to the data entity identifier, and
- retrieving the lineage data stored in association with the lineage data identifier;
- using one or more credentials associated with another user to determine a subset of input data entities another user has permissions to view;
- generating a response to the API request, wherein the response is a text-based representation of the generated lineage data in a structured schema language, wherein the response includes the data entity identifier for the generated data entity, and wherein the response includes at least one field for indicating the subset of input data entities that the generated data entity is dependent from, and a separate second field for indicating the one or more columns of the input data entities the column of the generated data entity is sourced from;
- and
- providing the response to the API request to expose a subset of the generated lineage data.
15 more elements
A method, comprising:
- receiving from a user of a client device, a request to execute a query on one or more input data entities;
- parsing the query to generate a query tree for the query where the query tree includes one or more nodes that correspond to operations defined by the query, wherein the one or more nodes of the query tree includes a root node and one or more leaf nodes;
- executing on a computing cluster, the query based on the query tree to generate at least one generated data entity, wherein the generated data entity is stored in association with a data entity identifier;
- generating a lineage data for the generated data entity by traversing the generated query tree starting from the root node to the one or more leaf nodes of the query tree, wherein the lineage data includes at least one or more dependencies for a column of the generated data entity relative to one or more columns of the input data entities;
- storing the generated lineage data in a database in association with a lineage identifier that is mapped to the data entity identifier;
- configuring an application programming interface (API) to expose the generated lineage data in the database;
- receiving, a second request for lineage data associated with the generated data entity via the API:
- retrieving the generated lineage data associated with the generated data entity from the database wherein retrieving the generated lineage data further comprises:
- obtaining the data entity identifier for the generated data entity,
- obtaining the lineage identifier that is mapped to the data entity identifier, and
- retrieving the lineage data stored in association with the lineage data identifier;
- using one or more credentials associated with another user to determine a subset of input data entities another user has permissions to view;
- generating a response to the API request, wherein the response is a text-based representation of the generated lineage data in a structured schema language, wherein the response includes the data entity identifier for the generated data entity, and wherein the response includes at least one field for indicating the subset of input data entities that the generated data entity is dependent from, and a separate second field for indicating the one or more columns of the input data entities the column of the generated data entity is sourced from;
- and
- providing the response to the API request to expose a subset of the generated lineage data.
12 more elements
A computer program product embodied in a non-transitory computer readable medium and comprising computer instructions for:
- receiving from a user of a client device, a request to execute a query on one or more input data entities;
- parsing the query to generate a query tree for the query where the query tree includes one or more nodes that correspond to operations defined by the query, wherein the one or more nodes of the query tree includes a root node and one or more leaf nodes;
- executing on a computing cluster, the query based on the query tree to generate at least one generated data entity, wherein the generated data entity is stored in association with a data entity identifier;
- generating a lineage data for the generated data entity by traversing the generated query tree starting from the root node to the one or more leaf nodes of the query tree, wherein the lineage data includes at least one or more dependencies for a column of the generated data entity relative to one or more columns of the input data entities;
- storing the generated lineage data in a database in association with a lineage identifier that is mapped to the data entity identifier;
- configuring an application programming interface (API) to expose the generated linage data in the database;
- receiving a second request for lineage data associated with the generated data entity via the API;
- retrieving the generated lineage associated with the generated data entity from the database wherein retrieving the generated lineage data further comprises:
- obtaining the data entity identifier for the generated data entity,
- obtaining the lineage identifier that is mapped to the data entity identifier, and
- retrieving the lineage data stored in association with the lineage data identifier;
- using one or more credentials associated with another user to determine a subset of input data entities another user has permissions to view;
- generating a response to the API request, wherein the response is a text-based representation of the generated lineage data in a structured schema language, wherein the response includes the data entity identifier for the generated data entity, and wherein the response includes at least one field for indicating the subset of input data entities that the generated data entity is dependent from, and a separate second field for indicating the one or more columns of the input data entities the column of the generated data entity is sourced from;
- and
- providing the response to the API request to expose a subset of the generated lineage data.
12 more elements
Dependencies are inferred from the claim text.