Training a machine learned model to determine relevance of items to a query using different sets of training data from a common domain

A method comprising:

• generating training data comprising a plurality of examples, each example comprising a query received by an online concierge system and an item with which a user of the online concierge system performed a specific interaction, wherein a label applied to each example of the training data indicates whether the specific interaction was performed with the item after the online concierge system received the query;
• generating a noisy subset of training data and a high-quality subset of training data based on a metric that defines a quality of training data, wherein the high-quality subset has a higher metric value than the noisy subset;
• initializing a machine-learned model comprising a network of a plurality of layers, the machine-learned model configured to receive a query and an item and to generate a predicted measure of relevance of the item to the query,
• for each of a plurality of the examples of the noisy subset of the training data:
• applying, by one or more processors, the machine-learned model to the query of the example of the noisy subset of the training data and to the item of the example of the noisy subset of the training data;
• backpropagating, in one or more iterations and by the one or more processors, one or more error terms obtained from one or more loss functions to update a set of parameters of the network, the backpropagating performed through the network and one or more of the error terms based on a difference between the label applied to the example of the noisy subset of the training data and a predicted measure of relevance of the item of the example of the training data and to the query of the example of the noisy subset of the training data;
• stopping, by the one or more processors, the backpropagation after the one or more loss functions satisfy one or more criteria;
• storing, in a computer readable storage medium, the set of parameters of the network that are updated in the one or more iterations;
• initializing the network to the stored set of parameters;
• for each of the plurality of the examples of the high-quality subset of the training data:
• applying, by the one or more processors, the machine-learned model to the query of the example of the high-quality subset of the training data and to the item of the example of the high-quality subset of the training data;
• backpropagating, by the one or more processors, one or more error terms obtained from one or more loss functions to generate a modified set of parameters of the network, the backpropagating performed through the network and one or more of the error terms based on a difference between a label applied to the example of the high-quality subset of the training data and a predicted measure of relevance of the item of the example of the high-quality subset of the training data and to the query of the example of the subset of the training data;
• stopping, by the one or more processors, the backpropagation after the one or more loss functions satisfy one or more criteria;
• and
• storing, in the computer readable storage medium, the modified set of parameters of the network trained from the subset of the training data as parameters of the machine-learned model.

A system comprising a computer readable storage medium and one or more processors, wherein the computer readable storage medium is configured to store code comprising instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:

• generate training data comprising a plurality of examples, each example comprising a query received by an online concierge system and an item with which a user of the online concierge system performed a specific interaction, wherein a label applied to each example of the training data indicates whether the specific interaction was performed with the item after the online concierge system received the query;
• generate a noisy subset of training data and a high-quality subset of training data based on a metric that defines a quality of training data, wherein the high-quality subset has a higher metric value than the noisy subset;
• initialize a machine-learned model comprising a network of a plurality of layers, the machine-learned model configured to receive a query and an item and to generate a predicted measure of relevance of the item to the query;
• for each of a plurality of the examples of the noisy subset of the training data:
• apply, by one or more processors, the machine-learned model to the query of the example of the noisy subset of the training data and to the item of the example of the noisy subset of the training data;
• backpropagate, in one or more iterations and by the one or more processors, one or more error terms obtained from one or more loss functions to update a set of parameters of the network, the backpropagating performed through the network and one or more of the error terms based on a difference between the label applied to the example of the noisy subset of the training data and a predicted measure of relevance of the item of the example of the training data and to the query of the example of the noisy subset of the training data;
• stop, by the one or more processors, the backpropagation after the one or more loss functions satisfy one or more criteria;
• store, in the computer readable storage medium, the set of parameters of the network that are updated in the one or more iterations;
• initialize the network to the stored set of parameters;
• for each of the plurality of the examples of the high-quality subset of the training data:
• apply, by the one or more processors, the machine-learned model to the query of the example of the high-quality subset of the training data and to the item of the example of the high-quality subset of the training data;
• backpropagate, by the one or more processors, one or more error terms obtained from one or more loss functions to generate a modified set of parameters of the network, the backpropagating performed through the network and one or more of the error terms based on a difference between a label applied to the example of the high-quality subset of the training data and a predicted measure of relevance of the item of the example of the high-quality subset of the training data and to the query of the example of the subset of the training data;
• stop, by the one or more processors, the backpropagation after the one or more loss functions satisfy one or more criteria;
• and
• store, in the computer readable storage medium, the modified set of parameters of the network trained from the subset of the training data as parameters of the machine-learned model.