Back to results
US 12,737,933 B2Granted
Text-based image generation using an image-trained text encoder
AssigneeAdobe Inc.
InventorsTobias Hinz, Ali Aminian, Hao Tan, Kushal Kafle, Oliver Wang, Jingwan Lu
Life of this patent
Estimating the term…
Abstract
A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining a text prompt and encoding, using a text encoder jointly trained with an image generation model, the text prompt to obtain a text embedding. Some embodiments generate, using the image generation model, a synthetic image based on the text embedding.
Claims18
A method for training a machine learning model, comprising:
- obtaining training data including a ground-truth image and a text prompt of the ground-truth image;
- generating, using an image generation model, a provisional image based on a provisional text embedding, wherein the provisional text embedding is generated based on the text prompt;
- and
- training a text encoder to generate text embeddings as input for generating images with the image generation model based on the provisional image and the ground-truth image, wherein the text encoder comprises a first plurality of layers and a second plurality of layers, wherein the text encoder is pre-trained prior to training the text encoder jointly with the image generation model, and wherein training the text encoder comprises fine-tuning the first plurality of layers and fixing the second plurality of layers.
1 more element
A method for image generation, comprising:
- obtaining a text prompt;
- encoding, using a text encoder jointly trained with an image generation model, the text prompt to obtain a text embedding, wherein the text encoder comprises a first plurality of layers and a second plurality of layers, wherein the text encoder is pre-trained prior to training the text encoder jointly with the image generation model, and wherein training the text encoder comprises fine-tuning the first plurality of layers and fixing the second plurality of layers;
- and
- generating, using the image generation model, a synthetic image based on the text embedding.
1 more element
A system for image generation, comprising:
- one or more processors;
- one or more memory components coupled with the one or more processors;
- a text encoder comprising text encoding parameters stored in the one or more memory components, a first plurality of layers, and a second plurality of layers, wherein the text encoder is trained to encode a text prompt to obtain a text embedding;
- and
- an image generation model comprising image generation parameters stored in the one or more memory components, the image generation model trained to generate a synthetic image based on the text embedding, wherein the text encoder is trained jointly with the image generation model based on an output of the image generation model, wherein the text encoder is pre-trained prior to training the text encoder jointly with the image generation model, and wherein training the text encoder comprises fine-tuning the first plurality of layers and fixing the second plurality of layers.
2 more elements
Dependencies are inferred from the claim text.