Automated switching of a strategy for selection of data elements to be labeled based on labeling progress

A system for outputting a data element to a user for an operation by the user to give a label to a plurality of data elements, the system comprising:

• a processor;
• and
• a memory coupled to the processor, wherein the memory comprises program instructions executable by the processor to cause the processor to perform a method comprising:
• employing classification learning of a model through active learning comprising:
• selecting a data element from a plurality of data elements, wherein each data element of the plurality of data elements is a document, by a first strategy, the first strategy being a strategy for selecting a data element which has been predicted with a low confidence level, the low confidence level at least partially subject to a prediction corresponding to one or more characteristics of the selected data element;
• outputting the selected data element so as for a user to give a label to the selected data element, thereby generating a labeled data element;
• learning the model with the labeled data element;
• predicting, subject to the model learning, the label for each unlabeled data element;
• assigning a confidence level of each prediction;
• and
• switching, automatically, between the first strategy and a second strategy depending on a progress degree of labeling by the user, the second strategy being a strategy for selecting a data element which has been predicted with a high confidence level, the high confidence level at least partially subject to the predicting;
• and
• stopping, automatically, at least partially subject to the predicting, all active learning processing comprising:
• determining that a matching rate between data element predictions and user data element labeling is lower than a potential accuracy rate of labeling by the user.

A computer program product for outputting a data element to a user for an operation by the user to give a label to a plurality of data elements, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• employing classification learning of a model through active learning comprising:
• selecting a data element from a plurality of data elements, wherein each data element of the plurality of data elements is a document, by first strategy, the first strategy being a strategy for selecting a data element which has been predicted with a low confidence level, the low confidence level at least partially subject to a prediction corresponding to one or more characteristics of the selected data element;
• outputting the selected data element so as for a user to give a label to the selected data element, thereby generating a labeled data element;
• learning the model with the labeled data element;
• predicting, subject to the model learning, the label for each unlabeled data element;
• assigning a confidence level of each prediction;
• and
• switching, automatically, between the first strategy and a second strategy depending on a progress degree of labeling by the user, the second strategy being a strategy for selecting a data element which has been predicted with a high confidence level, the high confidence level at least partially subject to the predicting;
• and
• stopping, automatically, at least partially subject to the predicting, all active learning processing comprising:
• determining that a matching rate between data element predictions and user data element labeling is lower than a potential accuracy rate of labeling by the user.