Compliance model training to classify landing page content that violates content item distribution guidelines

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

• receiving by the data processing apparatus training data that specify manual classifications of content items and feature values for each of the content items, the manual classification for each of the content items specifying whether the content item is a violating content item that violates content item distribution guidelines, the feature values specifying one or more characteristics of the content items and characteristics of landing pages to which the content items link;
• training by the data processing apparatus a compliance model using the training data, the compliance model being trained to classify an unclassified content item as a violating content item based on the feature values of the unclassified content item;
• determining by the data processing apparatus that the compliance model has an accuracy measure that meets a threshold accuracy measure;
• in response to determining that the accuracy measure for the compliance model meets the accuracy threshold, classifying by the data processing apparatus unclassified content items using the feature values for the unclassified content items;
• updating by the data processing apparatus the compliance model using the classifications of the unclassified content items;
• and
• preventing by the data processing a apparatus violating content items from being distributed:
• wherein classifying unclassified content items comprises classifying at least one unclassified content item as a suspicious content item, and further comprising:
• providing the suspicious content item to a rater for manual classification;
• receiving data specifying a manual classification of the suspicious content item;
• and
• updating the compliance model using the manual classification and the feature values of the suspicious content item.

A system comprising:

• a data store storing training data that specify manual classifications of content items and feature values for each of the content items, the manual classification for each of the content items specifying whether the content item is a violating content item that violates content item distribution guidelines, the feature values specifying one or more characteristics of the content items and characteristics of landing pages to which the content items link;
• and
• one or more computers, each having a memory, a processor, and code stored in the memory and executable in the processor, the one or more computers being operable to interact with the data store and to cause the processor to perform operations including:
• receiving the training data;
• training a compliance model using the training data, the compliance model being trained to classify an unclassified content item as a violating content item based on the feature values of the unclassified content item;
• determining that the compliance model has an accuracy measure that meets a threshold accuracy measure;
• in response to determining that the accuracy measure for the compliance model meets the accuracy threshold, classifying unclassified content items using the feature values for the unclassified content items;
• and
• providing data specifying the classifications of the unclassified content items;
• wherein the one or more computers are further operable to perform operations including:
• classifying at least one unclassified content item as a suspicious content item, and further comprising:
• providing the suspicious content item to a rater for manual classification;
• receiving data specifying a manual classification of the suspicious content item;
• and
• updating the compliance model using the manual classification and the feature values of the suspicious content item.