Remediation of data quality issues in computer databases

A system to remediate data quality issues in a consolidated data record that includes elements from a plurality of datasets, which comprises:

• a computer system comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
• source a first dataset from a data lake and a second dataset from the data lake;
• wherein sets of metadata respectively associated with the datasets are received with the first dataset and the second dataset;
• directly measure, based at least in part on dataset feed information, a first dataset flow quality value for the first dataset and a second dataset flow quality value for the second dataset for a consolidated data record resulting from combining a plurality of datasets;
• determine a predetermined dataset flow quality threshold value from the received metadata;
• responsive to said dataset flow quality value determinations, compare said dataset flow quality values with the predetermined dataset flow quality threshold value and, in response, conduct a consolidated data record dataset flow quality correction action for datasets having a dataset flow quality value beyond said dataset flow quality threshold value until the datasets having a dataset flow quality value beyond said dataset flow quality threshold are corrected to have dataset flow quality values not beyond the dataset flow quality threshold value;
• responsive to a determination that all of the first dataset and the second dataset have and/or are corrected to have dataset flow quality values not beyond the dataset flow quality threshold value, determine, based at least in part on dataset element health information, a plurality of first dataset health quality values for said first dataset and a plurality of second dataset health quality values for said second dataset, wherein the first dataset health quality values are based on different types of health quality attributes, wherein the second dataset health quality values are based on a different types of health quality attributes, wherein the first dataset health quality values are determined by computing a percentage of the records of the first dataset that match boundaries of a range of first expected health quality attribute values (HQAVs), wherein the second dataset health quality values are determined by computing a percentage of the records of the second dataset that match boundaries of a range of second expected HQAVs;
• determine a plurality of acceptable dataset health quality threshold values from the received metadata;
• and
• responsive to said dataset health quality value determinations, compare said health quality values with the determined acceptable dataset health quality threshold values and, in response, conduct a dataset health quality correction action for datasets having at least one dataset health quality value beyond an associated one of the determined acceptable dataset health quality threshold values that the at least one dataset health quality value is compared with.

A computer program product to remediate data quality issues in a consolidated data record that includes elements from a plurality of datasets, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:

• source, using said computer, a first dataset from a data lake and a second dataset from the data lake,
• wherein sets of metadata respectively associated with one of the datasets is received with the first dataset and the second dataset,
• directly measure, using said computer, based at least in part on dataset feed information, a first dataset flow quality value for the first dataset and a second dataset flow quality value for the second dataset for a consolidated data record resulting from combining a plurality of datasets;
• determine, using said computer, a predetermined dataset flow quality threshold value from the received metadata;
• responsive to said dataset flow quality value determinations, compare, using said computer, said dataset flow quality values with the predetermined dataset flow quality threshold value and, in response, conduct, using said computer, a consolidated data record dataset flow quality correction action for datasets having a dataset flow quality value beyond said dataset flow quality threshold value until the datasets having a dataset flow quality value beyond said dataset flow quality threshold are corrected to have dataset flow quality values not beyond the dataset flow quality threshold value,
• wherein said dataset feed information includes, for the datasets, dataset age thresholds and content sufficient for said computer to determine dataset ages,
• wherein said computer determines a dataset age of the first dataset and a dataset age of the second dataset,
• wherein the dataset age of the first dataset is an average age of elements of the first dataset, wherein the dataset age of the second dataset is an average age of elements of the second dataset,
• wherein said dataset flow quality value determinations are based, at least in part, on whether the respective determined dataset ages are greater than the associated dataset age thresholds,
• wherein said dataset flow quality values are determined, by said computer, to be beyond said dataset flow quality threshold in response to a determination that an associated dataset age is greater than the associated element age value threshold,
• wherein the associated dataset age is defined by a difference between a dataset flow query timestamp and an associated representative element timestamp;
• responsive to a determination that all of the first dataset and the second dataset have and/or are corrected to have dataset flow quality values not beyond the dataset flow quality threshold value, determine, using said computer, based at least in part on dataset element health information, a plurality of first dataset health quality values for said first dataset and a plurality of second dataset health quality values for said second dataset, wherein the first dataset health quality values are based on a different type of health quality attribute, wherein the second dataset health quality values are based on different types of health quality attributes, wherein the first dataset health quality values are determined by computing a percentage of the records of the first dataset that match boundaries of a range of first expected health quality attribute values (HQAVs), wherein the second dataset health quality values are determined by computing a percentage of the records of the second dataset that match boundaries of a range of second expected HQAVs;
• determine, using said computer, a plurality of acceptable dataset health quality threshold values from the received metadata;
• and
• responsive to said dataset health quality value determinations, compare, using said computer, said health quality values with an associated one of the determined acceptable dataset health quality threshold values and, in response, conduct a dataset health quality correction action for datasets having at least one dataset health quality value beyond an associated one of the determined acceptable dataset health quality threshold values that the at least one dataset health quality value is compared with.