Domain specific pre-training of cross modality transformer model
AssigneeInception Institute of Artificial Intelligence, Ltd.
InventorsDeng-Ping Fan, Mingchen Zhuge, Ling Shao
Life of this patent
Estimating the term…
Abstract
A transformer based vision-linguistic (VL) model and training technique uses a number of different image patches covering the same portion of an image, along with a text description of the image to train the model. The model and pre-training techniques may be used in domain specific training of the model. The model can be used for fine-grained image-text tasks in the fashion domain.
Claims24
A method of pre-training a predictive model in a fashion domain, the method comprising:
- receiving an input image of a fashion product and input text describing the fashion product;
- tokenizing the input text to generate a plurality of input text tokens;
- generating a plurality of patch groups of one or more image patches of the input image, each of the patch groups of one or more image patches covering a common portion of the input image;
- masking one or more of the image patches and the input text tokens;
- generating a training embedding from the image patches and the input text tokens;
- and
- training a cross modality transformer-based model to predict the masked one or more of the image patches and the input text tokens using the training embedding.
A non-transitory computer readable medium having stored thereon instructions, which when executed by one or more processors configure a computing system to perform a method of pre-training a predictive model in a fashion domain, the method comprising:
- receiving an input image of a fashion product and input text describing the fashion product;
- tokenizing the input text to generate a plurality of input text tokens;
- generating a plurality of patch groups of one or more image patches of the input image, each of the patch groups of one or more image patches covering a common portion of the input image;
- masking one or more of the image patches and the input text tokens;
- generating a training embedding from the image patches and the input text tokens;
- and
- training a cross modality transformer-based model to predict the masked one or more of the image patches and the input text tokens using the training embedding.
A computer system comprising:
- a processor for executing instructions;
- a memory storing the instructions, which when executed by the processor configure the computer system to perform a method of pre-training a predictive model in a fashion domain, the method comprising:
- receiving an input image of a fashion product and input text describing the fashion product;
- tokenizing the input text to generate a plurality of input text tokens;
- generating a plurality of patch groups of one or more image patches of the input image, each of the patch groups of one or more image patches covering a common portion of the input image;
- masking one or more of the image patches and the input text tokens;
- generating a training embedding from the image patches and the input text tokens;
- and
- training a cross modality transformer-based model to predict the masked one or more of the image patches and the input text tokens using the training embedding.
Dependencies are inferred from the claim text.