Multimodal Image Classifier using Textual and Visual Embeddings

A system, comprising:

• a data processing apparatus;
• a memory in data communication with the data processing apparatus and storing instructions that cause the data processing apparatus to perform operations comprising:
• for each image of a plurality of images:
• processing the image by a textual generator model to obtain a set of phrases that are descriptive of the content of the image, wherein each phrase is one or more terms;
• processing the set of phrases by a textual embedding model to obtain an embedding of predicted text for the image;
• and
• processing the image using an image embedding model to obtain an embedding of image pixels of the image;
• training a multimodal image classifier on the embeddings of predicted text for the images and the embeddings of image pixels for the images to produce, as output, labels of an output taxonomy to classify an image based on the image as input.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• for each image of a plurality of images:
• processing the image by a textual generator model to obtain a set of phrases that are descriptive of the content of the image, wherein each phrase is one or more terms;
• processing the set of phrases by a textual embedding model to obtain an embedding of predicted text for the image;
• and
• processing the image using an image embedding model to obtain an embedding of image pixels of the image;
• training a multimodal image classifier on the embeddings of predicted text for the images and the embeddings of image pixels for the images to produce, as output, labels of an output taxonomy to classify an image based on the image as input.