Machine-learnt field-specific standardization

A computer program product comprising computer-readable program code to be executed by one or more processors when retrieved from a non-transitory computer-readable medium, the program code including instructions to:

• train a machine-learning model to tokenize raw values and corresponding standardized values into raw token sequences and corresponding standardized token sequences;
• train the machine-learning model to learn standardization of each one of a plurality of specific data fields from token insertions and token substitutions that modify the raw token sequences to match the corresponding standardized token sequences;
• tokenize an input value into an input token sequence;
• determine, by the machine-learning model, a probability of inserting an insertion token after an insertion markable token in the input sequence, based on learning a standardization of one of the plurality of specific data fields which comprises inserting the insertion token after a class of the insertion markable token in the raw token sequences;
• determine whether the probability of inserting the insertion token exceeds a threshold;
• insert the insertion token, which is punctuation inserted after the insertion markable token in the input token sequence, in response to a determination that the probability of inserting the insertion token exceeds the threshold;
• determine, by the machine-learning model, a probability of substituting a substitution token, for a substitutable token that differs in location from the insertion markable token, in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises substituting the substitution token for a class of the substitutable token in the raw token sequences;
• determine whether the probability of substituting the substitution token exceeds another threshold;
• and
• substitute the substitution token, which is one of a nil character, a letter that is part of a word, or a digit that is part of a number, for the substitutable token in the input token sequence, in response to a determination that the probability of substituting the substitution token exceeds the other threshold.

A method for machine-learnt field-specific standardization, the method comprising:

• training a machine-learning model to tokenize raw values and corresponding standardized values into raw token sequences and corresponding standardized token sequences;
• training the machine-learning model to learn standardization of each one of a plurality of specific data fields from token insertions and token substitutions that modify the raw token sequences to match the corresponding standardized token sequences;
• tokenizing an input value into an input token sequence;
• determining, by the machine-learning model, a probability of inserting an insertion token after an insertion markable token in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises inserting the insertion token after a class of the insertion markable token in the raw token sequences;
• determining whether the probability of inserting the insertion token exceeds a threshold;
• inserting the insertion token, which is punctuation inserted after the insertion markable token in the input token sequence, in response to a determination that the probability of inserting the insertion token exceeds the threshold;
• determining, by the machine-learning model, a probability of substituting a substitution token, for a substitutable token that differs in location from the insertion markable token, in the input token sequence, based on learning a standardization of one of the plurality of specific data fields which comprises substituting the substitution token for a class of the substitutable token in the raw token sequences;
• determining whether the probability of substituting the substitution token exceeds another threshold;
• and
• substituting the substitution token, which is one of nil, a letter that is part of a word, or a digit that is part of a number, for the substitutable token in the input token sequence, in response to a determination that the probability of substituting the substitution token exceeds the other threshold.