Text-to-image system and method
AssigneeAdobe Inc.
InventorsRuiyi Zhang, Yufan Zhou, Tong Yu, Tong Sun, Rajiv Jain, Jiuxiang Gu, Christopher Alan Tensmeyer
Life of this patent
Estimating the term…
Abstract
Techniques and systems for training and/or implementing a text-to-image generation model are provided. A pre-trained multimodal model is leveraged for avoiding slower and more labor-intensive methodologies for training a text-to-image generation model. Accordingly, images without associated text (i.e., bare images) are provided to the pre-trained multimodal model so that it can produce generated text-image pairs. The generated text-image pairs are provided to the text-to-image generation model for training and/or implementing the text-to-image generation model.
Claims20
A method comprising:
- generating a plurality of generated text-image pairs by inputting a plurality of bare images to a pre-trained multimodal model having a text encoder and an image encoder implementing a multimodal embedding space in which cosine similarity of the plurality of bare images in the multimodal embedding space to a plurality of text, respectively, is used to generate the text-image pairs;
- training a text-to-image generation model using the plurality of generated text-image pairs, the text-to-image generation model trained as a generative adversarial network (GAN) having a generator and discriminator to train the generator to produce an image based upon text;
- and
- testing, during the training, the text being paired with the generated images for cosine similarity until a threshold value for the cosine similarity is achieved, the threshold value being at least 0.27.
A system comprising:
- a plurality of bare images;
- a pre-trained multimodal model that inputs the plurality of bare images and generates a plurality of generated text-image pairs, the pre-trained multimodal model having a text encoder and an image encoder implementing a multimodal embedding space in which cosine similarity of the plurality of bare images in the multimodal embedding space to a plurality of text, respectively, is used to generate the text-image pairs;
- and
- a text-to-image generation model trained as a generative adversarial network (GAN) having a generator and discriminator to train the generator to produce an image based upon text, the text-to-image generation model being trained using the plurality of bare images and the plurality of generated text-image pairs based, at least in part, on testing, during the training, the text being paired with the generated images for cosine similarity until a threshold value for the cosine similarity is achieved, the threshold value being at least 0.27.
A non-transitory computer-readable storage media storing instructions that, responsive to execution by a processing device, cause the processing device to perform operations comprising:
- inputting a plurality of bare images to a pre-trained multimodal model, the pre-trained model generating a plurality of generated text-image pairs, the pre-trained multimodal model having a text encoder and an image encoder implementing a multimodal embedding space in which cosine similarity of the plurality of bare images in the multimodal embedding space to a plurality of text, respectively, is used to generate the text-image pairs;
- and
- training a text-to-image generation model using the plurality of generated text-image pairs, the text-to-image generation model trained as a generative adversarial network (GAN) having a generator and discriminator to train the generator based, at least in part, on testing, during the training, the text being paired with the generated images for cosine similarity until a threshold value for the cosine similarity is achieved, the threshold value being at least 0.27.
Dependencies are inferred from the claim text.