Embedding Based Retrieval for Image Search

A computer-implemented method, the method comprising:

• obtaining, by a computing system comprising one or more processors, a training dataset, wherein the training dataset comprises a plurality of training image search queries and a plurality of training image search results, wherein each of the plurality of training image search results is associated with one or more respective training images search queries of the plurality of training image search queries;
• processing, by the computing system, a first training image search query of the plurality of training image search queries with a query embedding model to generate a first query embedding, wherein the first query embedding is descriptive of one or more features of the first training image search query;
• processing, by the computing system, a first image and a first landing page with a pair embedding model to generate a first pair numeric embedding, wherein the first pair numeric embedding is descriptive of one or more features of the first image and one or more features of the first landing page, wherein the first image and the first landing page are associated with a first training image search result of the plurality of training image search results, wherein the first training image search result is associated with the first training image search query;
• evaluating, by the computing system, a loss function that evaluates a difference between the first query embedding and the first pair numeric embedding;
• and
• adjusting, by the computing system, one or more parameters of the query embedding model based at least in part on the loss function.

A computing system, the system comprising:

• one or more processors;
• and
• one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing system to perform operations, the operations comprising:
• receiving an image search query;
• processing the image search query with a query embedding model to generate a search query embedding, wherein the search query embedding is descriptive of one or more features of the image search query, wherein the query embedding model was trained by:
• obtaining a training dataset, wherein the training dataset comprises a plurality of training image search queries and a plurality of training image search results;
• processing a first training image search query of the plurality of training image search queries with the query embedding model to generate a first query embedding;
• processing a first image and a first landing page with a pair embedding model to generate a first pair numeric embedding, wherein the first pair numeric embedding is descriptive of one or more features of the first image and one or more features of the first landing page, wherein the first image and the first landing page are associated with a first training image search result of the plurality of training image search results;
• evaluating a loss function that evaluates a difference between the first query embedding and the first pair numeric embedding;
• and
• adjusting one or more parameters of the query embedding model based at least in part on the loss function;
• determining a plurality of pair numeric embeddings are associated with the search query embedding, wherein the plurality of pair numeric embeddings are descriptive of a plurality of image-landing page pairs, wherein each of the plurality of pair numeric embeddings are associated with a respective image-landing page pair;
• and
• providing the plurality of image-landing page pairs as search results.

One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more computing devices, cause the one or more computing devices to perform operations, the operations comprising:

• obtaining a training dataset, wherein the training dataset comprises a plurality of training image search queries and a plurality of training image search results, wherein each of the plurality of training image search results is associated with one or more respective training images search queries of the plurality of training image search queries;
• processing a first training image search query of the plurality of training image search queries with an image search query embedding neural network to generate a first query embedding, wherein the first query embedding is descriptive of one or more features of the first training image search query;
• processing a first image and a first landing page with a pair embedding neural network to generate a first pair numeric embedding, wherein the first pair numeric embedding is descriptive of one or more features of the first image and one or more features of the first landing page, wherein the first image and the first landing page are associated with a first training image search result of the plurality of training image search results, wherein the first training image search result is associated with the first training image search query;
• evaluating a loss function that evaluates a difference between the first query embedding and the first pair numeric embedding;
• and
• jointly training the image search query embedding neural network and the pair embedding neural network based at least in part on the loss function.