Vision-and-language model training
AssigneeAmazon Technologies, Inc.
InventorsTarik Arici, Mehmet Saygin Seyfioglu, Ismail Baha Tutar, Tal Neiman
Life of this patent
Estimating the term…
Abstract
Systems and methods for improving training processes for image and text applications are described. A first set of embeddings may be generated based on a text input, and a second set of embeddings may be generated via a convolutional neural network (CNN), based on an input image. The first set of embeddings and the second set of embeddings may be utilized to generate a third set of embeddings including one or more placeholder values to be replaced. The placeholder values may be replaced based on predicted values, to reconstruct the input text and image.
Claims20
A computer-implemented method, comprising:
- obtaining a first set of embeddings, generated based on a first input, and a second set of embeddings generated based on a second input;
- generating, based at least in part on the first set of embeddings and the second set of embeddings, a third set of embeddings including one or more placeholder values associated with one or more values removed from at least the first set of embeddings or the second set of embeddings;
- and
- reconstructing at least a portion of the first input or the second input based on the third set of embeddings.
A computing system, comprising:
- a computing device processor;
- and
- a memory device including instructions that, when executed by the computing device processor, enable the computing system to:
- generate a first set of embeddings based on a text input;
- generate a second set of embeddings corresponding to an input image;
- associate the first set of embeddings with the second set of embeddings;
- generate, based at least in part on the first set of embeddings and the second set of embeddings, a third set of embeddings including one or more placeholder values associated with one or more values removed from the first set of embeddings and the second set of embeddings;
- predict one or more values corresponding to known values associated with the first set of embeddings and the second set of embeddings;
- and
- reconstruct at least one of the text input and the image input based, at least in part, on replacing the one or more placeholder values with the one or more predicted values.
A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to:
- obtain a first set of embeddings, generated based on a first input, and a second set of embeddings generated based on a second input;
- generate, based at least in part on the first set of embeddings and the second set of embeddings, a third set of embeddings including one or more placeholder values associated with one or more values removed from at least the first set of embeddings or the second set of embeddings;
- and
- reconstruct at least a portion of the first input or the second input based on the third set of embeddings.
Dependencies are inferred from the claim text.