Generating rules for data processing values of data fields from semantic labels of the data fields

A method for determining a data quality rule for values in one or more datasets, the method being implemented by a data processing system and comprising:

• profiling, by the data processing system, values stored in a field included in data records of one or more datasets;
• applying, by the data processing system, one or more classifiers to the profiled values;
• identifying, based on applying the one or more classifiers, one or more attributes indicative of a logical or syntactical characteristic for the values of the field, with each of the one or more attributes having a respective confidence level that is based on an output of each of the one or more classifiers;
• associating, with the field, one or more of the identified attributes for which the confidence level satisfies a threshold level;
• determining, based on the one or more attributes associated with the field, one or more constraints for values included in the field;
• and
• based on the determined one or more constraints, determining a data quality rule for the field of the dataset, the data quality rule indicating a relationship between values of the field and the one or more constraints.

A data processing system for determining a data quality rule for values in a field of a data record in a set of data records based on a label associated with the field, the label indicating the characteristics of the values of the field, the data processing system including:

• at least one processing device;
• and
• at least one memory in communication with the at least one processing device, the at least one memory storing instructions that, when executed by the at least one processing device, cause the at least one processing device to perform operations including:
• retrieving a label index that associates a label with a set of one or more fields in a data record, wherein the label identifies a type of information expected in each field of the set of one or more fields;
• accessing a data dictionary that associates the type of information indicated by the label with a set of attribute values representing requirements for values of the one or more fields associated with the label, the requirements including logical or syntactical characteristics of the values for the one or more fields;
• and
• for a field of a particular data record:
• identifying, by accessing the label index, a particular label associated with the field of the particular data record;
• retrieving, from the data dictionary, an attribute value for the particular label, the attribute value specifying a particular requirement for the field;
• and
• generating a data quality rule that, when executed, is configured to:
• validate whether a value of the field satisfies the particular requirement represented by the attribute value, and
• generate output data indicating whether the particular requirement is satisfied.

One or more non-transitory computer readable media storing instructions for determining a data quality rule for values in a field of a data record in a set of data records based on a label associated with the field, the label indicating the characteristics of the values of the field, the one or more instructions that, when executed by at least one processing device, are configured to cause the at least one processing device to perform operations including:

• retrieving a label index that associates a label with a set of one or more fields in a data record, wherein the label identifies a type of information expected in each field of the set of one or more fields;
• accessing a data dictionary that associates the type of information indicated by the label with a set of attribute values representing requirements for values of the one or more fields associated with the label, the requirements including logical or syntactical characteristics of the values for the one or more fields;
• and
• for a field of a particular data record:
• identifying, by accessing the label index, a particular label associated with the field of the particular data record;
• retrieving, from the data dictionary, an attribute value for the particular label, the attribute value specifying a particular requirement for the field;
• and
• generating a data quality rule that, when executed, is configured to:
• validate whether a value of the field satisfies the particular requirement represented by the attribute value, and
• generate output data indicating whether the particular requirement is satisfied.