Back to results
US 11,237,900 B2Granted
System for generating dataflow lineage information in a data network
AssigneeBank of America Corporation
InventorsAmitava Deb, Sandip Gopal Bhatwadekar, Chih-Chin Yang, Jovan Cenev
Life of this patent
Estimating the term…
Abstract
A system for aggregating dataflow lineage information is disclosed. The system receives one or more input data elements and determines a dataflow path for the one or more input data elements. The dataflow path includes at least a data storage node and a computation node. Then, the system identifies a lineage control value associated with the data storage node and a version control value associated with the computation node. The system generates an output lineage for the one or more input data elements by appending the lineage control value to the version control value.
Claims20
A system for aggregating dataflow lineage information, comprising:
- a plurality of data storage nodes, each of the data storage nodes being configured to store a node table, the node table comprising a plurality of node entries, wherein each of the node entries comprises:
- a node identifier identifying one of the plurality of data storage nodes;
- a node value associated with the identified date storage node;
- and
- a lineage control value associated with at least the node identifier and the node value;
- one or more computation nodes, each computation node being configured to perform a data transformation on an input data element;
- and
- a lineage server configured to store one or more version control tables for the one or more computation nodes, each version control table comprising a plurality of version control entries that each comprises:
- a computation name associated with a computation algorithm performed by the computation node;
- a version value associated with the computation algorithm;
- a timestamp associated with the version value;
- and
- a version control value associated with at least the computation name and the version value;
- the lineage server further configured to:
- receive one or more input data elements;
- determine a dataflow path for the one or more input data elements, the dataflow path comprising at least a first data storage node and a first computation node;
- identify, from a first node table stored in the first data storage node, a first node entry corresponding to the first data storage node;
- identify a first lineage control value in the first node entry;
- associate the first lineage control value to the one or more input data elements;
- identify a first version control table corresponding to the first computation node;
- identify, from the first version control table, a version control entry with a most recent timestamp;
- identify a version control value of the identified version control entry with the most recent timestamp;
- associate the identified version control value to the one or more input data elements;
- and
- generate an output lineage for the one or more input data elements by appending the first lineage control value to the identified version control value.
22 more elements
A non-transitory computer-readable medium comprising a logic for aggregating dataflow lineage information, the logic, when executed by one or more processors, instructing the one or more processors to:
- store a node table comprising a plurality of node entries, each of the node entries comprising:
- a node identifier identifying one of a plurality of data storage nodes;
- a node value associated with the identified date storage node;
- and
- a lineage control value associated with at least the node identifier and the node value;
- store one or more version control tables associated with one or more computation nodes, each version control table comprising a plurality of version control entries that each comprises:
- a computation name associated with a computation algorithm performed by the computation node;
- a version value associated with the computation algorithm;
- a timestamp associated with the version value;
- and
- a version control value associated with at least the computation name and the version value;
- receive one or more input data elements;
- determine a dataflow path for the one or more input data elements, the dataflow path comprising at least a first data storage node and a first computation node;
- identify, from the node table, a first node entry corresponding to the first data storage node;
- identify a first lineage control value in the first node entry;
- associate the first lineage control value to the one or more input data elements;
- identify a first version control table corresponding to the first computation node;
- identify, from the first version control table, a version control entry with a most recent timestamp;
- identify a version control value in the identified version control entry with the most recent timestamp;
- associate the identified version control value to the one or more input data elements;
- and
- generate an output lineage for the one or more input data elements by appending the first lineage control value to the identified version control value.
19 more elements
A method for aggregating dataflow lineage information, comprising:
- storing a node table comprising a plurality of node entries, each of the node entries comprising:
- a node identifier identifying one of a plurality of data storage nodes;
- a node value associated with the identified date storage node;
- and
- a lineage control value associated with at least the node identifier and the node value;
- storing one or more version control tables associated with one or more computation nodes, each version control table comprising a plurality of version control entries that each comprises:
- a computation name associated with a computation algorithm performed by the computation node;
- a version value associated with the computation algorithm;
- a timestamp associated with the version value;
- and
- a version control value associated with at least the computation name and the version value associated with the version value;
- receiving one or more input data elements;
- determining a dataflow path for the one or more input data elements, the dataflow path comprising at least a first data storage node and a first computation node;
- identifying, from the node table, a first node entry corresponding to the first data storage node;
- identifying a first lineage control value in the first node entry;
- associating the first lineage control value to the one or more input data elements;
- identifying a first version control table corresponding to the first computation node;
- identifying, from the first version control table, a version control entry with a most recent timestamp;
- identifying a version control value in the identified version control entry with the most recent timestamp;
- associating the identified version control value to the one or more input data elements;
- and
- generating an output lineage for the one or more input data elements by appending the first lineage control value to the identified version control value.
19 more elements
Dependencies are inferred from the claim text.