Mapping images to words for composed image retrieval

A system, comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving a query comprising:
• (i) an image depicting a scene, and (ii) a text prompt describing a context of the scene;
• processing the image, using a visual encoder, to generate a visual embedding of the image;
• processing the visual embedding of the image, using a mapping neural network, to generate one or more language tokens of the image;
• generating a plurality of language tokens of the text prompt;
• processing the language tokens of the image and text prompt, using a language encoder, to generate a language embedding of the query;
• and
• selecting, from a plurality of candidate images, one or more of the candidate images using the language embedding of the query.

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving a query comprising:
• (i) an image depicting a scene, and (ii) a text prompt describing a context of the scene;
• processing the image, using a visual encoder, to generate a visual embedding of the image;
• processing the visual embedding of the image, using a mapping neural network, to generate one or more language tokens of the image;
• generating a plurality of language tokens of the text prompt;
• processing the language tokens of the image and text prompt, using a language encoder, to generate a language embedding of the query;
• and
• selecting, from a plurality of candidate images, one or more of the candidate images using the language embedding of the query.