Learning unpaired multimodal feature matching for semi-supervised learning

A computer program product for learning multimodal feature matching, the computer program product comprising a non-transitory computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to perform a method comprising:

• training an image encoder with a triplet loss that pushes similar images together and dissimilar images apart to obtain encoded images;
• training a common classifier on the encoded images by using labeled images to learn text embeddings with corresponding labels;
• and
• training a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.

A computer processing system for learning multimodal feature matching, the computer processing system comprising:

• a memory device for storing program code;
• and
• a processor device operatively coupled to the memory device for running the program code to
• train an image encoder with a triplet loss that pushes similar images together and dissimilar images apart to obtain encoded images;
• train a common classifier on the encoded images by using labeled images to learn text embeddings with corresponding labels;
• and
• train a text encoder while keeping the common classifier in a fixed configuration by using learned text embeddings and corresponding labels for the learned text embeddings, wherein the text encoder is further trained to match a distance of predicted text embeddings which is encoded by the text encoder to a fitted Gaussian distribution on the encoded images.