Back to results
US 12,749,293 B2Granted
Modality specific learnable attention for multi-conditioned diffusion models
AssigneeAdobe Inc.
InventorsHareesh Ravi, Aashish Kumar Misraa, Ajinkya Gorakhnath Kale
Life of this patent
Estimating the term…
Abstract
A method, apparatus, non-transitory computer readable medium, and system for image generation include encoding a text prompt to obtain a text embedding. An image prompt is encoded to obtain an image embedding. Cross-attention is performed on the text embedding and then on the image embedding to obtain a text attention output and an image attention output, respectively. A synthesized image is generated based on the text attention output and the image attention output.
Claims19
A method comprising:
- obtaining a text prompt, an image prompt, and a noise input;
- encoding the text prompt to obtain a text embedding;
- encoding the image prompt to obtain an image embedding;
- generating, using an image generation model, an intermediate feature map based on the noise input;
- performing, using a text attention layer of the image generation model, cross-attention on the text embedding and the intermediate feature map to obtain a text attention output;
- performing, using an image attention layer of the image generation model, cross-attention on the image embedding and the same intermediate feature map provided to the text attention layer to obtain an image attention output;
- and
- generating, using a generator network of the image generation model, a synthesized image based on the text attention output and the image attention output.
5 more elements
A method of training a machine learning model, the method comprising:
- obtaining a training set including a training text prompt, a training image prompt, and a noise input;
- and
- training, using the training set, an image generation model to generate a synthesized image, the training comprising:
- encoding the training text prompt to obtain a text embedding;
- encoding the training image prompt to obtain an image embedding;
- generating, using the image generation model, an intermediate feature map based on the noise input;
- training a text attention layer of the image generation model to perform cross-attention on the text embedding and the intermediate feature map to obtain a text attention output;
- and
- training an image attention layer of the image generation model to perform cross-attention on the embedding and the same intermediate feature map provided to the text attention layer to obtain an image attention output.
6 more elements
An apparatus comprising:
- at least one processor;
- and
- at least one memory including instructions executable by the at least one processor to perform operations comprising:
- obtaining a text prompt, an image prompt, and a noise input;
- encoding the text prompt to obtain a text embedding;
- encoding the image prompt to obtain an image embedding;
- generating, using an image generation model, an intermediate feature map based on the noise input;
- performing, using a text attention layer of the image generation model, cross-attention on the text embedding and the intermediate feature map to obtain a text attention output;
- performing, using an image attention layer of the image generation model, cross-attention on the image embedding and the same intermediate feature map provided to the text attention layer to obtain an image attention output;
- and
- generating, using a generator network of the image generation model, a synthesized image based on the text attention output and the image attention output.
8 more elements
Dependencies are inferred from the claim text.