Back to results
US 11,231,978 B2Granted
API and streaming solution for documenting data lineage
AssigneeCapital One Services, LLC
InventorAnn Chapin
Life of this patent
Estimating the term…
Abstract
A system for tracing data lineage includes a non-transitory computer readable medium and a processor. The processor is configured to execute an application programming interface (API). The processor executes a first instance of the API to document a first data lineage in a first data transformation process. The processor executes a second instance of the API to document a second data lineage in a second data transformation process. The processor sends the first data lineage and the second data lineage for storage in the non-transitory computer readable medium.
Claims19
A system for tracing data lineage, comprising:
- memory storing a plurality of predetermined rules for documenting data lineage;
- and
- a processor, wherein the processor:
- receives a first input data of a first data transformation process;
- determines whether the first input data comprises a data error;
- responsive to determining the first input data comprises the data error, documents a first data lineage associated with the first data transformation process in a first document;
- responsive to determining the first input data does not comprise the data error:
- determines that the first data lineage associated with the first data transformation process meets a first predetermined rule;
- and
- documents the first data lineage in a second document according to the first predetermined rule;
- receives a second input data of a second data transformation process;
- compares the second input data to a predetermined threshold;
- and
- aborts at least a part of the second data transformation process when the second input data exceeds the predetermined threshold;
- wherein the first predetermined rule requires documenting the first data lineage when the first data lineage meets a threshold computed based on a plurality of data lineages;
- and
- wherein the threshold includes at least one of an average, mean, or median computed from the plurality of data lineages.
14 more elements
A method for tracing data lineage, comprising:
- receiving, at a processor, a first input data of a first data transformation process from a transformation system;
- determining whether the first input data comprises a data error;
- responsive to determining the first input data comprises the data error, documenting, by the processor, a first data lineage associated with the first data transformation process in a first document;
- responsive to determining the first input data does not comprise the data error:
- determining, by the processor, that the first data lineage associated with the first data transformation process meets a first predetermined rule;
- and
- documenting, by the processor, the first data lineage in a second document according to the first predetermined rule;
- receiving, at the processor, a second input data of a second data transformation process from the transformation system;
- comparing, by the processor, the second input data to a predetermined threshold;
- and
- sending, by a transceiver, an instruction to the transformation system to at least partially abort the second data transformation process when the second input data exceeds the predetermined threshold;
- wherein the first predetermined rule requires documenting the first data lineage when the first data lineage meets a threshold computed based on a plurality of data lineages;
- and
- wherein the threshold includes at least one of an average, mean, or median computed from the plurality of data lineages.
11 more elements
A non-transitory computer-readable medium comprising instructions that, when executed by a system cause the s stem to:
- receive a first input data of a first data transformation process from a transformation system;
- determine whether the first input data comprises a data error;
- responsive to determining the first input data comprises the data error, document a first data lineage associated with the first data transformation process in a first document;
- responsive to determining the first input data does not comprise the data error:
- determine that the first data lineage associated with the first data transformation process meets a first predetermined rule;
- and
- document the first data lineage in a second document according to the first predetermined rule;
- receive a second input data of a second data transformation process from the transformation system;
- compare the second input data to a predetermined threshold;
- and
- send an instruction to the transformation system to at least partially abort the second data transformation process when the second input data exceeds the predetermined threshold;
- wherein the first predetermined rule requires documenting the first data lineage when the first data lineage meets a threshold computed based on a plurality of data lineages;
- and
- wherein the threshold includes at least one of an average, mean, or median computed from the plurality of data lineages.
11 more elements
Dependencies are inferred from the claim text.