US 2025/0147958 A1Published application

Training a machine learned model to determine relevance of items to a query using different sets of training data from a common domain

AssigneeMaplebear Inc.

InventorsTaesik Na, Yuqing Xie, Tejaswi Tenneti, Haixun Wang

Life of this patent

Estimating the term…

Loading patent…

Abstract

An online concierge system maintains various items and an item embedding for each item. When the online concierge system receives a query for retrieving one or more items, the online concierge system generates an embedding for the query. The online concierge system trains a machine-learned model to determine a measure of relevance of an embedding for a query to item embeddings by generating training data of examples including queries and items with which users performed a specific interaction. The online concierge system generates a subset of the training data including examples satisfying one or more criteria and further trains the machine-learned model by application to the examples of the subset of the training data and stores parameters resulting from the further training as parameters of the machine-learned model.

Claims20

Dependent claims

A computer-implemented method, comprising:

receiving a query for retrieving one or more items;
applying a machine-learned model to identify a plurality of candidate items, wherein the machine-learned model is trained by:
generating training data comprising a plurality of examples, each example comprising a historical query received by an online system and an item with which a user of the online system performed a specific interaction, wherein a label applied to each example of the training data indicates whether the specific interaction was performed with the item after the online system received the historical query;
generating a noisy subset of training data and a high-quality subset of training data based on a metric that defines a quality of training data, wherein the high-quality subset has a higher metric value than the noisy subset;
initializing the machine-learned model of a plurality of layers, the machine-learned model configured to receive the historical query and the item of an example;
generating predicted measures of the items of the examples;
training the machine-learned model using the examples of the noisy subset of the training data to update a set of parameters of the machine-learned model through backpropagating error terms based on differences between the labels applied to the examples of the noisy subset of the training data and predicted measures of the examples of the noisy subset of the training data;
storing the set of parameters of the machine-learned model that are trained by the noisy subset as a first set of trained parameters;
training the machine-learned model using the examples of the high-quality subset of the training data to update the first set of trained parameters through backpropagating error terms based on differences between the labels applied to the examples of the high-quality subset of the training data and predicted measures of the examples of the high-quality subset of the training data;
storing updated set of parameters of the machine-learned model that are trained by the high-quality subset as a second set of trained parameters;
and
storing the second set of trained parameters trained from the high-quality subset of the training data as parameters of the machine-learned model;
and
returning one or more candidate items identified by the machine-learned model in response to the query.

11 more elements

A non-transitory computer-readable storage medium configured to store code comprising instructions, wherein the instructions, when executed by one or more processors, cause the one or more processors to:

receive a query for retrieving one or more items;
apply a machine-learned model to identify a plurality of candidate items, wherein the machine-learned model is trained by:
generating training data comprising a plurality of examples, each example comprising a historical query received by an online system and an item with which a user of the online system performed a specific interaction, wherein a label applied to each example of the training data indicates whether the specific interaction was performed with the item after the online system received the historical query;
generating a noisy subset of training data and a high-quality subset of training data based on a metric that defines a quality of training data, wherein the high-quality subset has a higher metric value than the noisy subset;
initializing the machine-learned model of a plurality of layers, the machine-learned model configured to receive the historical query and the item of an example;
generating predicted measures of the items of the examples;
training the machine-learned model using the examples of the noisy subset of the training data to update a set of parameters of the machine-learned model through backpropagating error terms based on differences between the labels applied to the examples of the noisy subset of the training data and predicted measures of the examples of the noisy subset of the training data;
storing the set of parameters of the machine-learned model that are trained by the noisy subset as a first set of trained parameters;
training the machine-learned model using the examples of the high-quality subset of the training data to update the first set of trained parameters through backpropagating error terms based on differences between the labels applied to the examples of the high-quality subset of the training data and predicted measures of the examples of the high-quality subset of the training data;
storing updated set of parameters of the machine-learned model that are trained by the high-quality subset as a second set of trained parameters;
and
storing the second set of trained parameters trained from the high-quality subset of the training data as parameters of the machine-learned model;
and
return one or more candidate items identified by the machine-learned model in response to the query.

11 more elements

A system comprising a computer readable storage medium and one or more processors, wherein the computer readable storage medium is configured to store code comprising instructions, wherein the instructions, when executed by the one or more processors, cause the one or more processors to:

receive a query for retrieving one or more items;
apply a machine-learned model to identify a plurality of candidate items, wherein the machine-learned model is trained by:
generating training data comprising a plurality of examples, each example comprising a historical query received by an online system and an item with which a user of the online system performed a specific interaction, wherein a label applied to each example of the training data indicates whether the specific interaction was performed with the item after the online system received the historical query;
generating a noisy subset of training data and a high-quality subset of training data based on a metric that defines a quality of training data, wherein the high-quality subset has a higher metric value than the noisy subset;
initializing the machine-learned model of a plurality of layers, the machine-learned model configured to receive the historical query and the item of an example;
generating predicted measures of the items of the examples;
training the machine-learned model using the examples of the noisy subset of the training data to update a set of parameters of the machine-learned model through backpropagating error terms based on differences between the labels applied to the examples of the noisy subset of the training data and predicted measures of the examples of the noisy subset of the training data;
storing the set of parameters of the machine-learned model that are trained by the noisy subset as a first set of trained parameters;
training the machine-learned model using the examples of the high-quality subset of the training data to update the first set of trained parameters through backpropagating error terms based on differences between the labels applied to the examples of the high-quality subset of the training data and predicted measures of the examples of the high-quality subset of the training data;
storing updated set of parameters of the machine-learned model that are trained by the high-quality subset as a second set of trained parameters;
and
storing the second set of trained parameters trained from the high-quality subset of the training data as parameters of the machine-learned model;
and
return one or more candidate items identified by the machine-learned model in response to the query.

11 more elements

Dependencies are inferred from the claim text.

Facts

Application: 19/012,864Filed 8 Jan 2025
Priority: 9 Feb 2022
Published: 8 May 2025
Term ends: Not estimatedThe source records don’t support an estimate.

Classification12 codes

G06F 16/24534
Query optimisation: query rewriting; Transformation
G06Q 10/08
Administration; Management: logistics, e.g. warehousing, loading or distribution; Inventory or stock management
G06F 16/2448
Query languages: for particular applications; for extensibility, e.g. user defined types
G06F 16/24578
With adaptation to user needs: using ranking

Family

Application17/668,358Continuation · filed 9 Feb 2022
Application19/012,864Continuation
US 12,222,937 B2Related patent
US 2023/0306023 A1Published application
US 2025/0147958 A1This patent

From the US filing history. Foreign family members aren’t listed.

Timeline

Citations

Open elsewhere

Building an agent or pulling many patents? Use the API or MCP server instead of scraping pages. Get an API key