Visual-semantic representation learning via multi-modal contrastive training

An apparatus comprising:

• an image encoder configured to encode images to produce encoded images;
• a text encoder configured to encode captions corresponding to the images to produce encoded text;
• and
• a training component configured to compute a multi-modal loss function based on the encoded images and the encoded text and to train the image encoder and the text encoder based on the multi-modal loss function, wherein the multi-modal loss function comprises at least one image loss term, at least one text loss term, and at least one cross-modal term.

A method of image search comprising:

• encoding an image using an image encoder to produce an encoded image;
• encoding text using a text encoder to produce encoded text, wherein the image encoder and the text encoder are jointly trained based on a multi-modal loss function comprising at least one image loss term, at least one text loss term, and at least one cross-modal term;
• and
• performing an image search based on the encoded image and the encoded text.