Global embedding learning from different modalities

An apparatus, comprising:

• a processor;
• memory coupled with the processor;
• and
• instructions stored in the memory and executable by the processor to cause the apparatus to:
• generating training data comprising training titles and training images, one or more training titles having a masked title portion and one or more training images have a masked image portion, the generating of the training data comprising:
• masking a portion of a first training image:
• masking a portion of a first training image;
• generating a predicted portion for the first training image based at least in part on masking the portion of the first training image and a title of the first training image;
• and
• generating a predicted listing category for the first training image based at least in part on the predicted portion for the first training image;
• train a machine learning model based on the training data;
• receive, via a first user interface associated with an online marketplace, a multi-modality request to retrieve a listing for an item, the multi-modality request comprising at least a first image and a first natural language text associated with the item;
• generate a first image token based at least in part on the first image and a first title token based at least in part on the first natural language text;
• generate, by the machine learning model, a first vector based on the first image token and the first title token;
• and
• cause presentation, via the first user interface associated with the online marketplace, of one or more listings for the multi-modality request based at least in part on a similarity metric between the first vector and each vector of a plurality of vectors associated with a plurality of listings.

A non-transitory computer-readable medium storing code, the code comprising instructions executable by a processor to perform operations comprising:

• generating training data comprising training titles and training images, one or more training titles having a masked title portion and one or more training images have a masked image portion, the generating of the training data comprising:
• masking a portion of a first training image;
• masking a portion of a first training image;
• generating a predicted portion for the first training image based at least in part on masking the portion of the first training image and a title of the first training image;
• and
• generating a predicted listing category for the first training image based at least in part on the predicted portion for the first training image;
• training a machine learning model based on the training data;
• receiving, via a first user interface associated with an online marketplace, a multi-modality request to retrieve a listing for an item, the multi-modality request comprising at least a first image and a first natural language text associated with the item;
• generating a first image token based at least in part on the first image and a first title token based at least in part on the first natural language text;
• generating, by the machine learning model, a first vector based on the first image token and the first title token;
• and
• causing presentation, via the first user interface associated with the online marketplace, of one or more listings for the multi-modality request based at least in part on a similarity metric between the first vector and each vector of a plurality of vectors associated with a plurality of listings.