Back to results
US 12,608,351 B2Granted
Database and data structure management systems
AssigneeTruist Bank
InventorsTufail Ahmed Khan, Pranjal Goswami, Di Hu, Changyong Wei
Life of this patent
Estimating the term…
Abstract
Systems and methods access, from one or more data storage locations, a dataset; perform data analysis on the dataset to detect one or more data quality characteristics each corresponding to at least one data quality dimension including timeliness, uniqueness, accuracy, completeness, validity, or consistency; evaluate the one or more data quality characteristics present in the dataset to identify one or more common patterns; and generate one or more data quality rule recommendations based on the identified one or more common patterns.
Claims20
A computing system, comprising:
- at least one processor;
- a communication interface communicatively coupled to the at least one processor;
- and
- a memory device storing executable code that, when executed, causes the at least one processor to:
- determine a plurality of data quality characteristics present in a dataset, respective ones of the data quality characteristics associated with a respective data quality dimension of a plurality of data quality dimensions;
- determine, based on the plurality of data quality characteristics, a plurality of common patterns in the dataset;
- generate, based on the plurality of common patterns and the plurality of data quality characteristics, a plurality of data quality rule recommendations for the dataset;
- generate, by at least two semantic type identification models including a regular expression model, a plurality of predetermined semantic types in the dataset;
- compute, based on the plurality of predetermined semantic types, a semantic type confidence score for each predetermined semantic type of the plurality of predetermined semantic types;
- generate, for each predetermined semantic type based on the semantic type confidence scores, a respective confidence label and a respective classification recommendation for the dataset;
- select, based on the semantic type confidence scores, a first predetermined semantic type of the plurality of predetermined semantic types;
- compute, for each data quality rule recommendation, a respective rule confidence score;
- determine that the rule confidence score for a first data quality rule recommendation of the plurality of data quality rule recommendations exceeds a predetermined threshold;
- implement, based on the determination that the rule confidence score for the first data quality rule recommendation exceeds the predetermined threshold, the first data quality rule recommendation to the dataset;
- implement, for the dataset, the confidence label and the classification recommendation associated with the selected first predetermined semantic type;
- collect usage pattern data for the implemented first data quality rule recommendation and the implemented classification recommendation associated with the selected first predetermined semantic type;
- compute a checksum of the dataset at a first time interval;
- compare the checksum of the dataset at the first time interval to a stored checksum of the dataset at a second time interval, the second time interval previous to the first time interval to determine a change in the dataset from the second time interval to the first time interval;
- and
- based on:
- (i) the collected usage pattern data for the implemented first data quality rule and the implemented classification recommendation associated with the selected first predetermined semantic type, and (ii) the determined change in the dataset, generate a new data quality rule recommendation.
18 more elements
A method comprising:
- determining a plurality of data quality characteristics present in a dataset, respective ones of the data quality characteristics associated with a respective data quality dimension of a plurality of data quality dimensions;
- determining, based on the plurality of data quality characteristics, a plurality of common patterns in the dataset;
- generating, based on the plurality of common patterns and the plurality of data quality characteristics, a plurality of data quality rule recommendations for the dataset;
- generating, by at least two semantic type identification models including a regular expression model, a plurality of predetermined semantic types in the dataset;
- computing, based on the plurality of predetermined semantic types, a semantic type confidence score for each predetermined semantic type of the plurality of predetermined semantic types;
- generating, for each predetermined semantic type based on the semantic type confidence scores, a respective confidence label and a respective classification recommendation for the dataset;
- selecting, based on the semantic type confidence scores, a first predetermined semantic type of the plurality of predetermined semantic types;
- computing, for each data quality rule recommendation, a respective rule confidence score;
- determining that the rule confidence score for a first data quality rule recommendation of the plurality of data quality rule recommendations exceeds a predetermined threshold;
- implementing, based on the determination that the rule confidence score for the first data quality rule recommendation exceeds the predetermined threshold, the first data quality rule recommendation to the dataset;
- implementing, for the dataset, the confidence label and the classification recommendation associated with the selected first predetermined semantic type;
- collecting usage pattern data for the implemented first data quality rule recommendation and the implemented classification recommendation associated with the selected first predetermined semantic type;
- computing a checksum of the dataset at a first time interval;
- comparing the checksum of the dataset at the first time interval to a stored checksum of the dataset at a second time interval, the second time interval previous to the first time interval to determine a change in the dataset from the second time interval to the first time interval;
- and
- based on:
- (i) the collected usage pattern data for the implemented first data quality rule and the implemented classification recommendation associated with the selected first predetermined semantic type, and (ii) the determined change in the dataset, generating a new data quality rule recommendation.
14 more elements
Dependencies are inferred from the claim text.