System and method for supervised contrastive learning for multi-modal tasks

A method comprising:

• obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text;
• and
• training a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:
• a first multi-modal representation loss based on the paired image-text pair;
• and
• a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair;
• wherein training the machine learning model comprises:
• determining word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs;
• determining visual features for the images of the paired image-text pairs and the unpaired image-text pairs;
• and
• implementing a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs.

An electronic device comprising:

• at least one memory configured to store instructions;
• and
• at least one processing device configured when executing the instructions to:
• obtain a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text;
• and
• train a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:
• a first multi-modal representation loss based on the paired image-text pair;
• and
• a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image.

A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to:

• obtain a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text;
• and
• train a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair:
• a first multi-modal representation loss based on the paired image-text pair;
• and
• a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image.