Loading patent…
AssigneeHewlett Packard Enterprise Development LP
InventorsAnnmary Justine KOOMTHANAM, Suparna Bhattacharya, Aalap Tripathy, Sergey Serebryakov, Martin Foltin, Paolo Faraboschi
Estimating the term…
Systems and methods are provide for automatically constructing data lineage representations for distributed data processing pipelines. These data lineage representations (which are constructed and stored in a central repository shared by the multiple data processing sites) can be used to among other things, clone the distributed data processing pipeline for quality assurance or debugging purposes. Examples of the presently disclosed technology are able to construct data lineage representations for distributed data processing pipelines by (1) generating a hash content value for universally identifying each data artifact of the distributed data processing pipeline across the multiple processing stages/processing sites of the distributed data processing pipeline; and (2) creating an data processing pipeline abstraction hierarchy for associating each data artifact to input and output events for given executions of given data processing stages (performed by the multiple data processing sites).
A method, comprising:
A non-transitory computer-readable medium storing instructions, which when executed by one or more processing resources of a second data processing site of a data processing pipeline, cause the one or more processing resources to:
A system comprising:
Dependencies are inferred from the claim text.