# US 12,183,062 B2 · System and method for supervised contrastive learning for multi-modal tasks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,183,062 B2 - Document ID: US12183062B2 - Status: Granted - Kind: B2 - Application number: 17/589,535 - Filed: 2022-01-31 - Published: 2023-08-03 - Granted: 2024-12-31 - Earliest claimed: 2022-01-31 - Assignees: Samsung Electronics Co., Ltd. - Applicants: Samsung Electronics Co., Ltd. - Inventors: Changsheng Zhao; Burak Uzkent; Yilin Shen; Hongxia Jin - Web page: https://patentgrep.com/patent/US12183062B2 ## CPC - G06V 10/811: {the classifiers operating on different input data, e.g. multi-modal recognition} - G06F 40/279: Recognition of textual entities - G06V 10/764: using classification, e.g. of video objects - G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting - G06V 10/778: Active pattern-learning, e.g. online learning of image or video features - G06V 10/806: {of extracted features} - G06V 10/82: using neural networks - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method includes obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, where each paired image-text pair and each unpaired image-text pair includes an image and a text. The method also includes training a machine learning model using the training data based on an optimization of a combination of losses. The losses include, for each paired image-text pair, (i) a first multi-modal representation loss based on the paired image-text pair and (ii) a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair. ## Claims ### Claim 1 A method comprising: obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and training a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair: a first multi-modal representation loss based on the paired image-text pair; and a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image. ### Claim 2 (depends on 1) The method of claim 1, further comprising: for each paired image-text pair, determining the second multi-modal representation loss by calculating a first loss associated with the first unpaired image-text pair and a second loss associated with the second unpaired image-text pair. ### Claim 3 (depends on 1) The method of claim 1, wherein the losses further include, for each paired image-text pair: a third multi-modal representation loss based on the paired image-text pair in which a representation of a first object in the image of the paired image-text pair is replaced with a representation of a second object that is of a same type as the first object; and a fourth multi-modal representation loss based on the paired image-text pair in which a representation of the first object in the image of the paired image-text pair is replaced with a representation of a third object that is of a different type than the first object. ### Claim 4 (depends on 1) The method of claim 1, wherein, in each paired image-text pair, at least one tag of the image matches at least one tag of the text. ### Claim 5 (depends on 1) The method of claim 1, wherein the machine learning model is trained in association with a downstream task comprising at least one of: visual question answering (VQA), graph question answering (GQA), referring expressions, caption generation, and image retrieval. ### Claim 6 (depends on 1) The method of claim 1, wherein the first multi-modal representation loss comprises a cross-entropy loss or an object detection loss. ### Claim 7 A method comprising: obtaining a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and training a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair: a first multi-modal representation loss based on the paired image-text pair; and a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs includes either the image or the text of the paired image-text pair; wherein training the machine learning model comprises: determining word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs; determining visual features for the images of the paired image-text pairs and the unpaired image-text pairs; and implementing a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs. ### Claim 8 An electronic device comprising: at least one memory configured to store instructions; and at least one processing device configured when executing the instructions to: obtain a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and train a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair: a first multi-modal representation loss based on the paired image-text pair; and a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein each of the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image. ### Claim 9 (depends on 8) The electronic device of claim 8, wherein the at least one processing device is further configured when executing the instructions to: for each paired image-text pair, determine the second multi-modal representation loss by calculating a first loss associated with the first unpaired image-text pair and a second loss associated with the second unpaired image-text pair. ### Claim 10 (depends on 8) The electronic device of claim 8, wherein the losses further include, for each paired image-text pair: a third multi-modal representation loss based on the paired image-text pair in which a representation of a first object in the image of the paired image-text pair is replaced with a representation of a second object that is of a same type as the first object; and a fourth multi-modal representation loss based on the paired image-text pair in which a representation of the first object in the image of the paired image-text pair is replaced with a representation of a third object that is of a different type than the first object. ### Claim 11 (depends on 8) The electronic device of claim 8, wherein, in each paired image-text pair, at least one tag of the image matches at least one tag of the text. ### Claim 12 (depends on 8) The electronic device of claim 8, wherein, to train the machine learning model, the at least one processing device is configured when executing the instructions to: determine word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs; determine visual features for the images of the paired image-text pairs and the unpaired image-text pairs; and implement a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs. ### Claim 13 (depends on 8) The electronic device of claim 8, wherein the at least one processing device is configured when executing the instructions to train the machine learning model in association with a downstream task comprising at least one of: visual question answering (VQA), graph question answering (GQA), referring expressions, caption generation, and image retrieval. ### Claim 14 (depends on 8) The electronic device of claim 8, wherein the first multi-modal representation loss comprises a cross-entropy loss or an object detection loss. ### Claim 15 A non-transitory machine-readable medium containing instructions that when executed cause at least one processor of an electronic device to: obtain a batch of training data including multiple paired image-text pairs and multiple unpaired image-text pairs, wherein each paired image-text pair and each unpaired image-text pair comprises an image and a text; and train a machine learning model using the training data based on an optimization of a combination of losses, wherein the losses include, for each paired image-text pair: a first multi-modal representation loss based on the paired image-text pair; and a second multi-modal representation loss based on two or more unpaired image-text pairs, selected from among the multiple unpaired image-text pairs, wherein the two or more unpaired image-text pairs comprise (i) a first unpaired image-text pair in which the text of the paired image-text pair is replaced with another text and (ii) a second unpaired image-text pair in which the image of the paired image-text pair is replaced with another image. ### Claim 16 (depends on 15) The non-transitory machine-readable medium of claim 15, further containing instructions that when executed cause the at least one processor to: for each paired image-text pair, determine the second multi-modal representation loss by calculating a first loss associated with the first unpaired image-text pair and a second loss associated with the second unpaired image-text pair. ### Claim 17 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the losses further include, for each paired image-text pair: a third multi-modal representation loss based on the paired image-text pair in which a representation of a first object in the image of the paired image-text pair is replaced with a representation of a second object that is of a same type as the first object; and a fourth multi-modal representation loss based on the paired image-text pair in which a representation of the first object in the image of the paired image-text pair is replaced with a representation of a third object that is of a different type than the first object. ### Claim 18 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein, in each paired image-text pair, at least one tag of the image matches at least one tag of the text. ### Claim 19 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the instructions that when executed cause the at least one processor to train the machine learning model comprise instructions that when executed cause the at least one processor to: determine word embeddings for the texts of the paired image-text pairs and the unpaired image-text pairs; determine visual features for the images of the paired image-text pairs and the unpaired image-text pairs; and implement a transformer network using the word embeddings and the visual features to learn multi-modal representations of the paired image-text pairs and the unpaired image-text pairs. ### Claim 20 (depends on 15) The non-transitory machine-readable medium of claim 15, wherein the first multi-modal representation loss comprises a cross-entropy loss or an object detection loss.