Back to results
US 12,613,842 B2Granted
Database and data structure management systems
AssigneeTruist Bank
InventorsTufail Ahmed Khan, Pranjal Goswami, Di Hu, Changyong Wei
Life of this patent
Estimating the term…
Abstract
Systems and methods access, from one or more data storage locations, a dataset; perform data analysis on the dataset to detect one or more data quality characteristics each corresponding to at least one data quality dimension including timeliness, uniqueness, accuracy, completeness, validity, or consistency; evaluate the one or more data quality characteristics present in the dataset to identify one or more common patterns; and generate one or more data quality rule recommendations based on the identified one or more common patterns.
Claims19
A computing system for database and data structure cleanup, comprising:
- at least one processor;
- a communication interface communicatively coupled to the at least one processor;
- and
- a memory device storing executable code that, when executed, causes the at least one processor to:
- access, from one or more data storage locations, at least one dataset;
- compute a checksum of each dataset;
- determine a plurality of quality characteristics of the at least one dataset, respective ones of the plurality of data quality characteristics associated with a respective data quality dimension of a plurality of data quality dimensions;
- determine, based on the plurality of data quality characteristics and the checksums, a plurality of common patterns in the at least one dataset;
- generate a plurality of data quality rule recommendations based the plurality of common patterns;
- classify, by a regular expression model based on a plurality of regular expressions, data in the at least one dataset into a plurality of predetermined semantic types;
- optimize the plurality of data quality rule recommendations based on the classifications, the optimization comprising:
- computing, for each data quality rule recommendation, a respective rule confidence score;
- determining that the rule confidence score for a first data quality rule of a first data quality rule recommendation of the plurality of data quality rule recommendations exceeds a predetermined threshold;
- and
- based on the determination that the rule confidence score for the first data quality rule recommendation exceeds the predetermined threshold, initiating implementation of the first data quality rule in association with at least one the dataset;
- and
- run a data cleanup process, comprising:
- applying the first data quality rule to the at least one dataset to determine whether data in the at least one dataset correlates conforms to the first data quality rule, by:
- determining, by the regular expression model based on the plurality of regular expressions, a semantic type of a name of each column in the at least one dataset;
- and
- determining, by a machine learning model, a semantic type of data in each field of the at least one dataset;
- in response to a determination, based at least in part on the semantic types of the names of each column and the semantic types of data in each field, that a first portion of the data in the at least one dataset conforms to the first data quality rule, retain the first portion of the data in the dataset;
- and
- in response to a determination, based at least in part on the semantic types of the names of each column and the semantic types of data in each field, that a second portion of the data in the at least one dataset does not conform to the first data quality rule, apply the first data quality rule to filter the second portion of the data to remove the second portion of the data from the at least one dataset.
21 more elements
A method for database and data structure cleanup comprising:
- accessing, from one or more data storage locations, at least one dataset;
- computing a checksum of each dataset;
- determining a plurality of quality characteristics of the at least one dataset, respective ones of the plurality of data quality characteristics associated with a respective data quality dimension of a plurality of data quality dimensions;
- determining, based on the plurality of data quality characteristics and the checksums, a plurality of common patterns in the at least one dataset;
- generating a plurality of data quality rule recommendations based on the plurality of common patterns;
- classifying, by a regular expression model based on a plurality of regular expressions, data in the at least one dataset into a plurality of predetermined semantic types;
- optimizing the plurality of data quality rule recommendations based on the classifications, the optimization comprising:
- computing, for each data quality rule recommendation, a respective rule confidence score;
- determining that the rule confidence score for a first data quality rule of a first data quality rule recommendation of the plurality of data quality rule recommendations exceeds a predetermined threshold;
- and
- based on the determination that the rule confidence score for the first data quality rule exceeds the predetermined threshold, initiating implementation of the first data quality rule in association with the at least one dataset;
- and
- running a data cleanup process, comprising:
- applying the first data quality rule to the at least one dataset to determine whether data in the at least one dataset correlates conforms to the first data quality rule, by:
- determining, by the regular expression model based on the plurality of regular expressions, a semantic type of a name of each column in the at least one dataset;
- and
- determining, by a machine learning model, a semantic type of data in each field of the at least one dataset;
- in response to a determination, based at least in part on the semantic types of the names of each column and the semantic types of data in each field, that a first portion of the data in the dataset conforms to the first data quality rule, retaining the first portion of the data in the at least one dataset;
- and
- in response to a determination, based at least in part on the semantic types of the names of each column and the semantic types of data in each field, that a second portion of the data in the dataset does not conform to the first data quality rule, apply the first data quality rule to filter the second portion of the data to removing the second portion of the data from the at least one dataset.
17 more elements
Dependencies are inferred from the claim text.