Sampling content using machine learning to identify low-quality content

A method comprising:

• obtaining a risk score determined for a content item provided by a user of an online system for display to users of the online system, the risk score indicating a likelihood of the content item violating a content policy of the online system;
• obtaining a probability density value indicating a relative frequency of the risk score among risk scores of content items provided by the users of the online system;
• obtaining an impression count of the content item indicating a number of times the content item has been presented to the users of the online system;
• determining a sampling weight for the content item based on the probability density value and the impression count of the content item;
• determining whether to select the content item for inclusion in a sampled subset of content items based on the sampling weight;
• and
• responsive to determining to select the content item for inclusion in the sampled subset of content items, sending the content item to a client device for presentation in a quality review interface.

A computer program product comprising a non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to:

• obtain a risk score determined for a content item provided by a user of an online system for display to users of the online system, the risk score indicating a likelihood of the content item violating a content policy of the online system;
• obtain a probability density value indicating a relative frequency of the risk score among risk scores of content items provided by the users of the online system;
• obtain an impression count of the content item indicating a number of times the content item has been presented to the users of the online system;
• determine a sampling weight for the content item based on the probability density value and the impression count of the content item, wherein the determined sampling weight is inversely proportional to the obtained probability density value;
• determine whether to select the content item for inclusion in a sampled subset of content items based on the sampling weight;
• responsive to determining to select the content item for inclusion in the sampled subset of content items, send the content item to a client device for presentation in a quality review interface.