Data lineage summarization

Software stored in a non-transitory form on a computer-readable medium, for managing lineage information in a computing system, the software including instructions for causing a computing system to:

• receive, over an input device or port, an identification of a directed graph that includes one or more data nodes that represent data elements, and includes directed links between data nodes that represent respective lineage relationships between data elements that are received and produced, respectively, by a computation during execution of the computation, or directed links between data nodes and data transformation nodes that represent computations that transform data elements, where the directed links between data nodes and data transformation nodes represent respective lineage relationships between a computation and a data element to be received or produced by the computation during execution of the computation;
• and
• compute, using at least one processor, display information based on paths over directed links in the directed graph and hierarchical container relationships among nodes in the directed graph, and using the display information to generate a data lineage diagram as a visual representation of at least portions of the directed graph, the computing including:
• traversing nodes along the paths over directed links in the directed graph and determining one or more of the nodes of the directed graph to exclude from the data lineage diagram based at least in part on any tag identifiers or tag values associated with traversed nodes, where at least one of the traversed nodes is associated with one or more tag identifiers of a plurality of tag identifiers, and at least one tag identifier of the plurality of tag identifiers has a plurality of possible tag values;
• receiving designation of interest for a plurality of the nodes of the directed graph;
• generating one or more summary objects included in the display information for one or more remaining nodes not included in the plurality of nodes of interest and not excluded based on the traversing, a first summary object of the one or more summary objects including summary information based on a first path over directed links in the directed graph between a first node of interest and a second node of interest;
• and
• generating one or more container objects included in the display information according to the hierarchical container relationships among the nodes, where each container object is able to be represented both by both a collapsed visual representation in the data lineage diagram, and an expanded visual representation in the data lineage diagram that contains visual representations of at least one of:
• (1) one or more data transformation nodes or data nodes, (2) one or more summary objects, or (3) one or more container objects;
• wherein generating the first summary object includes traversing the first path between the first node of interest and the second node of interest to identify one or more remaining nodes that are not designated as being of interest along the first path, and forming the summary information for the identified one or more remaining nodes.

A computing system for managing lineage information in a computing system, the computing system including:

• an input device or port for receiving an identification of a directed graph that includes one or more data nodes that represent data elements, and includes directed links between data nodes that represent respective lineage relationships between data elements that are received and produced, respectively, by a computation during execution of the computation, or directed links between data nodes and data transformation nodes that represent computations that transform data elements, where the directed links between data nodes and data transformation nodes represent respective lineage relationships between a computation and a data element to be received or produced by the computation during execution of the computation;
• and
• at least one processor for computing display information based on paths over directed links in the directed graph and hierarchical container relationships among nodes in the directed graph, and using the display information to generate a data lineage diagram as a visual representation of at least portions of the directed graph, the computing including:
• traversing nodes along the paths over directed links in the directed graph and determining one or more of the nodes of the directed graph to exclude from the data lineage diagram based at least in part on any tag identifiers or tag values associated with traversed nodes, where at least one of the traversed nodes is associated with one or more tag identifiers of a plurality of tag identifiers, and at least one tag identifier of the plurality of tag identifiers has a plurality of possible tag values;
• receiving designation of interest for a plurality of the nodes of the directed graph;
• generating one or more summary objects included in the display information for one or more remaining nodes not included in the plurality of nodes of interest and not excluded based on the traversing, a first summary object of the one or more summary objects including summary information based on a first path over directed links in the directed graph between a first node of interest and a second node of interest;
• and
• generating one or more container objects included in the display information according to the hierarchical container relationships among the nodes, where each container object is able to be represented both by both a collapsed visual representation in the data lineage diagram, and an expanded visual representation in the data lineage diagram that contains visual representations of at least one of:
• (1) one or more data transformation nodes or data nodes, (2) one or more summary objects, or (3) one or more container objects;
• wherein generating the first summary object includes traversing the first path between the first node of interest and the second node of interest to identify one or more remaining nodes that are not designated as being of interest along the first path, and forming the summary information for the identified one or more remaining nodes.

A computing system for managing lineage information in a computing system, the computing system including:

• means for receiving an identification of a directed graph that includes one or more data nodes that represent data elements, and includes directed links between data nodes that represent respective lineage relationships between data elements that are received and produced, respectively, by a computation during execution of the computation, or directed links between data nodes and data transformation nodes that represent computations that transform data elements, where the directed links between data nodes and data transformation nodes represent respective lineage relationships between a computation and a data element to be received or produced by the computation during execution of the computation;
• and
• means for computing, using at least one processor, display information based on paths over directed links in the directed graph and hierarchical container relationships among nodes in the directed graph, and using the display information to generate a data lineage diagram as a visual representation of at least portions of the directed graph, the computing including:
• traversing nodes along the paths over directed links in the directed graph and determining one or more of the nodes of the directed graph to exclude from the data lineage diagram based at least in part on any tag identifiers or tag values associated with traversed nodes, where at least one of the traversed nodes is associated with one or more tag identifiers of a plurality of tag identifiers, and at least one tag identifier of the plurality of tag identifiers has a plurality of possible tag values;
• receiving designation of interest for a plurality of the nodes of the directed graph;
• generating one or more summary objects included in the display information for one or more remaining nodes not included in the plurality of nodes of interest and not excluded based on the traversing, a first summary object of the one or more summary objects including summary information based on a first path over directed links in the directed graph between a first node of interest and a second node of interest;
• and
• generating one or more container objects included in the display information according to the hierarchical container relationships among the nodes, where each container object is able to be represented both by both a collapsed visual representation in the data lineage diagram, and an expanded visual representation in the data lineage diagram that contains visual representations of at least one of:
• (1) one or more data transformation nodes or data nodes, (2) one or more summary objects, or (3) one or more container objects;
• wherein generating the first summary object includes traversing the first path between the first node of interest and the second node of interest to identify one or more remaining nodes that are not designated as being of interest along the first path, and forming the summary information for the identified one or more remaining nodes.