Ad copy quality detection and scoring

A computing system for determining a cost for displaying a particular online advertisement based on a quality of the particular online advertisement, the computing system comprising a processor and computer storage memory with computer-executable instructions stored thereon that, when executed by the processor, are configured to:

• store a plurality of online advertisements in an advertising data store;
• generate an editorial webpage that is configured to present at least one of the plurality of online advertisements and receive human annotation data associated with the at least one of the plurality of online advertisements, the human annotation data including (1) a description of an overall quality of the at least one of the plurality of online advertisements and (2) an indication of a specific textual defect associated with a portion of text included in the at least one of the plurality of online advertisements, wherein the human annotation data is stored in association with the at least one of the plurality of online advertisements in the advertising data store;
• use a machine-learning algorithm to automatically determine a quality score for the particular online advertisement, wherein the machine-learning algorithm is trained with the human annotation data and with data from a language model that comprises n-grams taken from text in the plurality of online advertisements, wherein the machine-learning algorithm receives an input comprising an output from the language model that indicates a frequency with which one or more n-grams from the particular online advertisement match n-grams from the language model, and wherein the machine-learning algorithm generates the quality score for the particular online advertisement based on the received input;
• determine, based on the quality score, whether a surcharge is to be charged for displaying the particular online advertisement at a publisher webpage;
• and
• deliver the particular online advertisement to the publisher webpage.

A method for determining, using a computing device having a processor and a memory, a cost for displaying a particular online advertisement based on a quality of the particular online advertisement, the method comprising:

• storing, in the memory of the computing device, a plurality of online advertisements;
• retrieving at least one of the plurality of online advertisements from the memory of the computing device for presentation at an editorial webpage that is configured to receive human annotation data associated with the at least one of the plurality of online advertisements, the human annotation data including (1) a description of an overall quality of the at least one of the plurality of online advertisements and (2) an indication of a specific textual defect associated with a portion of text included in the at least one of the plurality of online advertisements;
• receiving the human annotation data from the editorial webpage;
• storing, in the memory of the computing device, the human annotation data in association with the at least one of the plurality of online advertisements;
• generating a language model using text from the plurality of online advertisements, wherein the language model allows an n-gram's frequency of occurrence within the plurality of online advertisements to be determined;
• at the processor of the computing device, generating a final model based on a combination of the language model and the human annotation data;
• using the final model to automatically calculate a quality score for the particular online advertisement, wherein the quality score is based, at least in part, on a frequency with which one or more n-grams occurring in the particular online advertisement match n-grams occurring in the plurality of online advertisements;
• selecting the particular online advertisement for delivery to a publisher webpage;
• and
• determining, based on the quality score, whether a surcharge is to be charged for displaying the particular online advertisement at the publisher webpage.