Random and active learning for classifier training

A method comprising:

• receiving a content item from a client device operated by a user of an online system;
• generating a priority score for the content item, the priority score representing a priority for labeling the content item for training a classification model;
• comparing the priority score of the content item to a threshold priority score;
• responsive to the priority score exceeding the threshold priority score, providing the content item to an authority for labeling;
• receiving a label from the authority for the content item;
• training the classification model with training data including the first set of content items, the second set of content items, and the received labels;
• assigning the content item to a bin of a histogram based on the priority score of the content item, the histogram comprising a plurality of bins;
• and
• adjusting the threshold priority score based on a set of content items stored in the plurality of bins of the histogram, the set of content items comprising the content item.

A non-transitory, computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:

• receive a plurality of content items from one or more client devices operated by one or more users of an online system;
• randomly select a first set of the plurality of content items;
• actively select a second set of the plurality of content items by:
• applying a classification model to each content item of a subset of the plurality of content items to determine a classification of each content item,
• determining a confidence score of the trained model for the classification for each content item of the subset of the plurality of content items, each generated confidence score representing a confidence of the classification model in the classification, and
• selecting a second set of the plurality of content items based on the confidence scores of the subset of content items;
• provide the first set of content items and the second set of content items to an authority for labeling;
• receive labels from the authority for the first set of content items and the second set of content items;
• and
• train the classification model with training data including the first set of content items, the second set of content items, and the received labels.

A non-transitory, computer-readable medium comprising instructions that, when executed by a processor, cause the processor to:

• receive a content item from a client device operated by a user of an online system;
• generate a priority score for the content item, the priority score representing a priority for labeling the content item for training a classification model;
• compare the priority score of the content item to a threshold priority score;
• responsive to the priority score exceeding the threshold priority score, provide the content item to an authority for labeling;
• receive a label from the authority for the content item;
• train the classification model with training data including the first set of content items, the second set of content items, and the received labels;
• assign the content item to a bin of a histogram based on the priority score of the content item, the histogram comprising a plurality of bins;
• and
• adjust the threshold priority score based on a set of content items stored in the plurality of bins of the histogram, the set of content items comprising the content item.