System and method for active learning to detect personally identifying information

A system, comprising:

• a processor configured to:
• identify one or more data sets by comparing each of the one or more data sets to a quality threshold;
• create training data set from the identified one or more data sets;
• train a machine learning (ML) model using the training data set based on labeled features in a first portion of a textual document;
• determine a measure of uncertainty regarding accuracy of the labeling, for other portions of the textual document;
• select a second portion of the textual document for labeling based on the respective measures of uncertainty and based on a determination that the second portion of the textual document contributes to further training of the ML model and accounts for a variable cost and contribution of labeling by an authoritative source;
• label features of the second portion of the textual document using the ML model;
• compare the features of the second portion labeled by the ML model to the features of the second portion labeled by the authoritative source;
• and
• further train the ML model based on the comparison.

A non-transitory computer readable medium encoded with a computer program that comprise instructions to cause a processor to:

• identify one or more data sets by comparing each of the one or more data sets to a quality threshold;
• create training data set from the identified one or more data sets;
• train a machine learning (ML) model using the training data set based on labeled features in a first portion of a textual document;
• determine a measure of uncertainty regarding accuracy of the labeling, for other portions of the textual document;
• select a second portion of the textual document for labeling based on the respective measures of uncertainty and based on a determination that the second portion of the textual document contributes to further training of the ML model and accounts for a variable cost and contribution of labeling by an authoritative source;
• label features of the second portion of the textual document using the ML model;
• comparing the features of the second portion labeled by the ML model to the features of the second portion labeled by the authoritative source;
• and
• further train the ML model based on the comparison.