Back to results
US 10,481,961 B1Granted
API and streaming solution for documenting data lineage
AssigneeCapital One Services, LLC
InventorAnn Chapin
Life of this patent
Estimating the term…
Abstract
A system for tracing data lineage includes a non-transitory computer readable medium and a processor. The processor is configured to execute an application programming interface (API) to perform the following. The processor instantiates a first instance of the API to document a first data lineage in a first data transformation process. The processor instantiates a second instance of the API to document a second data lineage in a second data transformation process. The processor sends the first data lineage and the second data lineage for storage in the non-transitory computer readable medium.
Claims16
A system for tracing data lineage, comprising:
- a non-transitory computer readable medium storing a plurality of predetermined rules for documenting data lineage;
- and
- a processor in communication with a transformation system that performs a plurality of data transformation processes, the processor configured to receive at least one of input data and output data of each data transformation process from the transformation system, the processor configured to execute an application programming interface (API) to perform the following:
- instantiate a first instance of the API to perform the following:
- receive first input data of a first data transformation process submitted to the transformation system, the first data transformation process identifiable by a first transformation process identifier;
- compare the first input data to a first predetermined threshold;
- send a first instruction to the transformation system to at least partially abort the first data transformation process based on the comparison between the first input data and the first predetermined threshold;
- determine whether a first data lineage associated with the first data transformation process meets a first predetermined rule;
- and
- document the first data lineage associated with the first data transformation process according to the first predetermined rule, the first data lineage including the first transformation process identifier;
- instantiate a second instance of the API to perform the following:
- receive second input data of a second data transformation process submitted to the transformation system, the second data transformation process identifiable by a second transformation process identifier;
- compare the second input data to a second predetermined threshold;
- send a second instruction to the transformation system to proceed with the second data transformation process based on the comparison between the second input data and the second predetermined threshold;
- determine whether a second data lineage associated with the second data transformation process meets a second predetermined rule, wherein the second predetermined rule requires documenting the second data lineage when the second data lineage meets a rule threshold relative to a derived metric computed based on a plurality of data lineages of a plurality of data processes, and wherein the derived metric includes at least one of average, mean and median computed from the plurality of data lineages;
- and
- document the second data lineage associated with the second data transformation process according to the second predetermined rule, the second data lineage including the second transformation process identifier;
- and
- send the first data lineage and the second data lineage for storage in the non-transitory computer readable medium, wherein the first data lineage tracks information related to data associated with the first data transformation process, and the second data lineage tracks information related to data associated with the second data transformation process.
16 more elements
A method for tracing data lineage, comprising:
- receiving, by a processor, at least one of input data and output data of each data transformation process from a transformation system;
- executing, by the processor, an application programming interface (API) to perform the following:
- instantiating a first instance of the API to perform the following:
- receive first input data of a first data transformation process submitted to the transformation system, the first data transformation process identifiable by a first transformation process identifier;
- compare the first input data to a first predetermined threshold;
- send a first instruction to the transformation system to at least partially abort the first data transformation process based on the comparison between the first input data and the first predetermined threshold;
- determine whether a first data lineage associated with the first data transformation process meets a first predetermined rule;
- and
- document the first data lineage associated with the first data transformation process according to the first predetermined rule, the first data lineage including the first transformation process identifier;
- instantiating a second instance of the API to perform the following:
- receive second input data of a second data transformation process submitted to the transformation system, the second data transformation process identifiable by a second transformation process identifier;
- compare the second input data to a second predetermined threshold;
- send a second instruction to the transformation system to proceed with the second data transformation process based on the comparison between the second input data and the second predetermined threshold;
- determine whether a second data lineage associated with the second data transformation process meets a second predetermined rule, wherein the second predetermined rule requires documenting the second data lineage when the second data lineage meets a rule threshold relative to a derived metric computed based on a plurality of data lineages of a plurality of data processes, and wherein the derived metric includes at least one of average, mean and median computed from the plurality of data lineages;
- and
- document the second data lineage associated with the second data transformation process according to the second predetermined rule, the second data
- lineage including the second transformation process identifier;
- and
- sending the first data lineage and the second data lineage for storage in a non-transitory computer readable medium, wherein the first data lineage tracks information related to data associated with the first data transformation process, and the second data lineage tracks information related to data associated with the second data transformation process.
16 more elements
Dependencies are inferred from the claim text.