Back to results
US 11,146,475 B2Granted
System for performing an integrated data quality control in a data network
AssigneeBank of America Corporation
InventorsAmitava Deb, Sandip Gopal Bhatwadekar, Chih-Chin Yang, Jovan Cenev
Life of this patent
Estimating the term…
Abstract
A system for performing an integrated data quality control is disclosed. The system determines a dataflow path for one or more input data elements. The, the system performs a lineage control check, a timeliness control check, and a variation control check on the dataflow path. If the dataflow path integrated scoring of the three controls is sufficient, the system determines that data related to the dataflow path is fit for use. If the dataflow path fails any one of the three checks, the system determines that data related to the dataflow path is not fit for use.
Claims17
A system for determining a data quality of a dataflow, comprising:
- a memory configured to store:
- a plurality of output lineages corresponding to a plurality of dataflow paths, each of the output lineages comprising a plurality of control values associated with the plurality of network nodes of a corresponding dataflow path, each of the control values associated with at least a node value and a timestamp;
- and
- one or more processors communicatively coupled to the memory and configured to:
- identify a first dataflow path, the first dataflow path comprising at least a first network node and a second network node;
- generate a first dataflow graph for the first dataflow path;
- in response to determining that the first dataflow graph matches one of one or more pre-learned dataflow graphs, determine that the first dataflow path passes a lineage control check;
- determine a first output lineage corresponding to the first dataflow path, the first output lineage comprising at least a first control value associated with the first network node and a second control value associated with the second network node;
- determine a latency between the first network node and the second network node based on the first control value and the second control value;
- in response to determining that the determined latency is lower than a pre-determined latency threshold, determine that the first dataflow path passes a timeliness control check;
- and
- in response to determining that the first dataflow path passes the lineage control check and the timeliness control check, determine that the first dataflow path and the first output lineage are fit for use.
9 more elements
A non-transitory computer-readable medium comprising a logic for determining a data quality of a dataflow, the logic, when executed by one or more processors, instructing the one or more processors to:
- store a plurality of output lineages corresponding to a plurality of dataflow paths, each of the output lineages comprising a plurality of control values associated with the plurality of network nodes of a corresponding dataflow path, each of the control values associated with at least a node value and a timestamp;
- identify a first dataflow path, the first dataflow path comprising at least a first network node and a second network node;
- generate a first dataflow graph for the first dataflow path;
- in response to determining that the first dataflow graph matches one of one or more pre-learned dataflow graphs, determine that the first dataflow path passes a lineage control check;
- determine a first output lineage corresponding to the first dataflow path, the first output lineage comprising at least a first control value associated with the first network node and a second control value associated with the second network node;
- determine a latency between the first network node and the second network node based on the first control value and the second control value;
- in response to determining that the determined latency is lower than a pre-determined latency threshold, determine that the first dataflow path passes a timeliness control check;
- and
- in response to determining that the first dataflow path passes the lineage control check and the timeliness control check, determine that the first dataflow path and the first output lineage are fit for use.
6 more elements
A method for performing a latency control for determining data quality of a dataflow, comprising:
- storing a plurality of output lineages corresponding to a plurality of dataflow paths, each of the output lineages comprising a plurality of control values associated with the plurality of network nodes of a corresponding dataflow path, each of the control values associated with at least a node value and a timestamp;
- identifying a first dataflow path, the first dataflow path comprising at least a first network node and a second network node;
- generating a first dataflow graph for the first dataflow path;
- in response to determining that the first dataflow graph matches one of one or more pre-learned dataflow graphs, determining that the first dataflow path passes a lineage control check;
- determining a first output lineage corresponding to the first dataflow path, the first output lineage comprising at least a first control value associated with the first network node and a second control value associated with the second network node;
- determining a latency between the first network node and the second network node based on the first control value and the second control value;
- in response to determining that the determined latency is lower than a pre-determined latency threshold, determining that the first dataflow path passes a timeliness control check;
- and
- in response to determining that the first dataflow path passes the lineage control check and the timeliness control check, determining that the first dataflow path and the first output lineage are fit for use.
6 more elements
Dependencies are inferred from the claim text.