Back to results
US 12,602,358 B2Granted
Database and data structure management systems
AssigneeTruist Bank
InventorsTufail Ahmed Khan, Pranjal Goswami, Di Hu, Changyong Wei
Life of this patent
Estimating the term…
Abstract
Systems and methods access, from one or more data storage locations, a dataset; perform data analysis on the dataset to detect one or more data quality characteristics each corresponding to at least one data quality dimension including timeliness, uniqueness, accuracy, completeness, validity, or consistency; evaluate the one or more data quality characteristics present in the dataset to identify one or more common patterns; and generate one or more data quality rule recommendations based on the identified one or more common patterns.
Claims20
A computing system for providing a database and data structure management, comprising:
- at least one processor;
- a communication interface communicatively coupled to the at least one processor;
- and
- a memory device storing executable code that, when executed, causes the at least one processor to:
- generate an interactive portal to manage a data quality rules engine;
- access a plurality of data quality rules that are generated based on identified common patterns in a dataset, the plurality of data quality rules corresponding to a plurality of detected data quality characteristics, each data quality characteristic corresponding to at least one data quality dimension of a plurality of data quality dimensions comprising timeliness, accuracy, uniqueness, completeness, validity, and consistency;
- compute a respective rule confidence score for each of the plurality of data quality rules;
- determine the rule confidence score for a first data quality rule of the plurality of data quality rules exceeds a predetermined threshold;
- request, based on the determination that the rule confidence score for the first data quality rule exceeds the predetermined threshold, input via the interactive portal regarding whether the first data quality rule should be implemented;
- in response to receiving input via the interactive portal that the first data quality rule should be implemented, initiating implementation of the first data quality rule by establishing a new data filter, the implementing comprising:
- receiving new data for potential inclusion into the dataset;
- applying the first data quality rule to the new data to determine whether the new data correlates to the dataset, by:
- determining, by a regular expression model based on a plurality of regular expressions, a semantic type of a name of each column in the new data;
- and
- determining, by a machine learning model, a semantic type of data in each field of the new data;
- in response to a determination, based at least in part on the semantic types of the names of each column in the new data and the semantic types of data in each field in the new data, that a first portion of the new data correlates to the dataset, store the first portion of the new data within the dataset;
- and
- in response to a determination, based at least in part on the semantic types of the names of each column in the new data and the semantic types of data in each field in the new data, that a second portion of the new data does not correlate to the dataset, apply the first data quality rule to filter the second portion of the new data and refrain from storing the second portion of the new data within the dataset.
15 more elements
A method comprising:
- accessing a plurality of data quality rules that are generated based on identified common patterns in a dataset, the plurality of data quality rules corresponding to a plurality of detected data quality characteristics, each data quality characteristic corresponding to at least one data quality dimension of a plurality of data quality dimensions comprising timeliness, accuracy, uniqueness, completeness, validity, and consistency;
- compute a respective rule confidence score for each of the plurality of data quality rules;
- determine the rule confidence score for a first data quality rule of the plurality of data quality rules exceeds a predetermined threshold;
- determining, based on the determination that the rule confidence score for the first data quality rule exceeds the threshold, that the first data quality rule should be implemented;
- initiating implementation of the first data quality rule by establishing a new data filter, the implementing comprising:
- receiving new data for potential inclusion into the dataset;
- applying the first data quality rule to the new data to determine whether the new data correlates to the dataset, by:
- determining, by a regular expression model based on a plurality of regular expressions, a semantic type of a name of each column in the new data;
- and
- determining, by a machine learning model, a semantic type of data in each field of the new data;
- in response to a determination, based at least in part on the semantic types of the names of each column in the new data and the semantic types of data in each field in the new data, that a first portion of the new data correlates to the dataset, store the first portion of the new data within the correlated dataset;
- and
- in response to a determination, based at least in part on the semantic types of the names of each column in the new data and the semantic types of data in each field in the new data, that a second portion of the new data does not correlate to the datasets, apply the first data quality rule to filter the second portion of the new data and reject the second portion of the new data and avoid associating the second portion of the new data with the dataset.
10 more elements
Dependencies are inferred from the claim text.