# US 12,217,191 B2 · Learning unpaired multimodal feature matching for semi-supervised learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,191 B2
- Document ID: US12217191B2
- Status: Granted
- Kind: B2
- Application number: 17/109,550
- Filed: 2020-12-02
- Published: 2022-06-02
- Granted: 2025-02-04
- Earliest claimed: 2020-12-02
- Assignees: International Business Machines Corporation
- Applicants: International Business Machines Corporation
- Inventors: Subhajit Chaudhury; Daiki Kimura; Gakuto Kurata; Ryuki Tachibana
- Web page: https://patentgrep.com/patent/US12217191B2

## CPC

- G06N 3/08: Learning methods
- G06N 5/04: Inference or reasoning models
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computer-implemented method is provided for learning multimodal feature matching. The method includes training an image encoder to obtain encoded images. The method further includes training a common classifier on the encoded images by using labeled images. The method also includes training a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings. The text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.

## Claims

### Claim 1

A computer-implemented method for learning multimodal feature matching comprising:
training an image encoder with a triplet loss that pushes similar images together and dissimilar images apart to obtain encoded images;
training a common classifier on the encoded images by using labeled images to learn text embeddings with corresponding labels; and
training a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising training the common classifier by using the labeled images with the image encoder and labeled text with the text encoder.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein the text encoder is trained to simultaneously optimize a cross entropy with the common classifier and a KL divergence between the fitted Gaussian distribution and the predicted text embeddings.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the common classifier is trained without paired data.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein the common classifier is trained using a cross-entropy loss.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein a total loss is computed as a sum of a loss corresponding to the common classifier and a result of multiplying a hyperparameter by a loss corresponding to the image encoder.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, further comprising minimizing a Kullback-Liebler divergence between the fitted Gaussian distribution and the learned text embeddings with the corresponding labels.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising performing semi-supervised learning on a common embedding space.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the text encoder maps pre-trained text embeddings to a common latent representation together with the image embeddings to enable cross-modal tasks.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, further comprising extracting the text embeddings by a pre-trained text embedding model applied to training text.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein the method is performed by a text captioning system that captions input images with output text descriptions.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising controlling an automobile for collision avoidance, responsive to at least one of the output text descriptions indicating an impending collision.

### Claim 13 (depends on 1)

The computer-implemented method of claim 1, wherein the triplet loss pushes similar ones of the encoded images together and dissimilar ones of the encoded images apart.

### Claim 14 (depends on 1)

The computer-implemented method of claim 1, wherein training the text encoder further comprises mapping, using the common classifier, the learned text embeddings to sample clusters to classify the learned text embeddings into respective ones of a plurality of classes.

### Claim 15 (depends on 1)

The computer-implemented method of claim 1, wherein the text encoder is trained such that a cross-entropy loss with a classifier in the fixed configuration and multi-variate Gaussian loss are simultaneously optimized.

### Claim 16 (depends on 1)

The computer-implemented method of claim 1, further comprising adding a random adjective to each of the corresponding labels for text distribution variation.

### Claim 17

A computer program product for learning multimodal feature matching, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:
training an image encoder with a triplet loss that pushes similar images together and dissimilar images apart to obtain encoded images;
training a common classifier on the encoded images by using labeled images to learn text embeddings with corresponding labels; and
training a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.

### Claim 18 (depends on 17)

The computer program product of claim 17, wherein the method further comprises training the common classifier by using the labeled images with the image encoder and labeled text with the text encoder.

### Claim 19 (depends on 17)

The computer program product of claim 17, wherein the text encoder is trained to simultaneously optimize a cross entropy with the common classifier and a KL divergence between the fitted Gaussian distribution and the predicted text embeddings.

### Claim 20 (depends on 17)

The computer program product of claim 17, wherein the common classifier is trained without paired data.

### Claim 21 (depends on 17)

The computer program product of claim 17, wherein the common classifier is trained using a cross-entropy loss.

### Claim 22 (depends on 17)

The computer program product of claim 17, wherein a total loss is computed as a sum of a loss corresponding to the common classifier and a result of multiplying a hyperparameter by a loss corresponding to the image encoder.

### Claim 23 (depends on 17)

The computer program product of claim 17, wherein the method further comprises minimizing a Kullback-Liebler divergence between the fitted Gaussian distribution and the learned text embeddings with the corresponding labels.

### Claim 24 (depends on 17)

The computer program product of claim 17, wherein the method is performed by a text captioning system that captions input images with output text descriptions.

### Claim 25

A computer processing system for learning multimodal feature matching, the computer processing system comprising:
a memory device for storing program code; and
a processor device operatively coupled to the memory device for running the program code to
train an image encoder with a triplet loss that pushes similar images together and dissimilar images apart to obtain encoded images;
train a common classifier on the encoded images by using labeled images to learn text embeddings with corresponding labels; and
train a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.