Outcome estimation models trained using regression and ranking techniques

A method comprising:

• receiving a request for content items;
• selecting a set of eligible content items responsive to the request;
• for each content item in the set of eligible content items,
• receiving feature values for the content item;
• computing an estimated click-through likelihood using the feature values of the content item and an outcome estimation model that has been trained using regression-based training and ranking-based training;
• and
• computing a selection score for the content item;
• selecting, from the set of eligible content items, one or more content items having a selection score that exceeds a selection threshold;
• and
• providing data that cause presentation of the selected one or more content items at a user device,
• wherein the outcome estimation model is trained by:
• initializing feature weights of the outcome estimation model, each feature weight representing a relative importance of a feature value for predicting an outcome;
• selecting, as training data with which the feature weights are to be updated, one of a data record and a record pair, each data record being associated with a set of feature values and a reference outcome, the data record being selected with a first probability and the record pair being selected with a second probability;
• updating the feature weights of the outcome estimation model using the selected training data;
• determining whether a stop condition has been met;
• in response to determining that the stop condition has not been met, repeating the selecting, the updating, and the determining;
• and
• in response to determining that the stop condition has been met, determining that the outcome estimation model is trained.

A non-transitory computer storage medium encoded with a computer program, the program comprising instructions that when executed by data processing apparatus cause the data processing apparatus to perform operations comprising:

• initializing feature weights of an outcome estimation model of the type that uses both regression error measures and ranking error measures, the regression error measures specifying error measures associated with training outcomes for respective data records, the ranking error measures specifying error measures associated with training outcomes for respective record pairs, each feature weight representing a relative importance of a feature value for predicting an outcome;
• selecting, as training data with which the feature weights are to be updated, one of a data record and a record pair, each data record being associated with a set of feature values and a reference outcome, the data record being selected with a first probability and the record pair being selected with a second probability;
• updating the feature weights of the outcome estimation model using the selected training data;
• determining whether a stop condition has been met;
• in response to determining that the stop condition has not been met, repeating the selecting, the updating, and the determining;
• and
• in response to determining that the stop condition has been met, determining that the outcome estimation model is trained.

A system comprising:

• a data store storing:
• data records representing content items, each data record specifying feature values and reference outcomes associated with a particular content item, the feature values for each content item representing characteristics of the content item, each data record being indexed according to the reference outcome that is associated with the content item being represented by the data record, and
• respective bids that are respectively associated with each of the content items, each respective bid specifying a maximum amount that will be paid for distribution of the content with which the respective bid is associated;
• and
• a data processing apparatus coupled to the data store, the data processing apparatus comprising one or more computers configured to train an outcome estimation model using the data records, the outcome estimation model having at least a first pre-specified minimum prediction quality measure using regression-based prediction quality measures and at least a second pre-specified minimum prediction quality measure using ranking-based prediction quality measures,
• wherein the data processing apparatus is further configured to compute a selection score using an output of the outcome estimation model and select, using the selection score, a set of content items for presentation in response to a request for content items.