Machine Learning Classification with Model Quality Prediction

A computer-implemented method comprising:

• selecting, by a computing device, a plurality of subsets from a corpus of training data, wherein the corpus of training data includes a plurality of entries, each entry having an observation respectively associated with a ground truth category of the observation, and wherein each of the plurality of subsets contains a different number of entries;
• for each particular subset of the plurality of subsets, the computing device:
• (i) dividing the particular subset into a training portion and a validation portion, (ii) training a machine learning based classifier with the training portion, wherein the classifier maps input observations into respective categories, (iii) providing the validation portion as input to the classifier as trained, and (iv) based on how entries of the validation portion are mapped to the categories, determining a respective precision for the particular subset;
• and
• based on the numbers of entries in the plurality of subsets and the respective precisions, recommending, by the computing device, a particular subset to use in training the classifier for a production environment.

An article of manufacture including a non-transitory computer-readable medium, having stored thereon program instructions that, upon execution by a computing system, cause the computing system to perform operations comprising:

• selecting a plurality of subsets from a corpus of training data, wherein the corpus of training data includes a plurality of entries, each entry having an observation respectively associated with a ground truth category of the observation, and wherein each of the plurality of subsets contains a different number of entries;
• for each particular subset of the plurality of subsets:
• (i) dividing the particular subset into a training portion and a validation portion, (ii) training a machine learning based classifier with the training portion, wherein the classifier maps input observations into respective categories, (iii) providing the validation portion as input to the classifier as trained, and (iv) based on how entries of the validation portion are mapped to the categories, determining a respective precision for the particular subset;
• and
• based on the numbers of entries in the plurality of subsets and the respective precisions, recommending a particular subset to use in training the classifier for a production environment.