Generative image filling using a reference image
AssigneeAdobe Inc.
InventorsYuqian Zhou, Krishna Kumar Singh, Zhe Lin, Qing Liu, Zhifei Zhang, Sohrab Amirghodsi, Elya Shechtman, Jingwan Lu
Life of this patent
Estimating the term…
Abstract
Embodiments include systems and methods for generative image filling based on text and a reference image. In one aspect, the system obtains an input image, a reference image, and a text prompt. Then, the system encodes the reference image to obtain an image embedding and encodes the text prompt to obtain a text embedding. Subsequently, a composite image is generated based on the input image, the image embedding, and the text embedding.
Claims20
A method comprising:
- obtaining an input image, a reference image, and a text prompt;
- encoding, using an image encoder, the reference image to obtain an image embedding;
- encoding, using a text encoder, the text prompt to obtain a text embedding;
- and
- generating, using an image generation model, a composite image based on the input image, the image embedding, and the text embedding.
A method comprising:
- obtaining an input image and a reference image;
- inserting the reference image into the input image to obtain a combined input image;
- encoding, using an image encoder, the reference image to obtain an image embedding;
- and
- generating, using an image generation model, a composite image based on the combined input image and the image embedding.
An apparatus comprising:
- at least one memory;
- at least one processor coupled to the at least one memory, wherein the processor is configured to execute instructions stored in the at least one memory;
- an image encoder configured to encode a reference image to obtain an image embedding;
- a text encoder configured to encode a text prompt to obtain a text embedding;
- and
- an image generation model configured to generate a composite image based on an input image, the image embedding, and the text embedding.
The system of aspect 15, wherein:
- the image generation model comprises a diffusion model.
The system of aspect 15, wherein:
- the image generation model comprises a self-attention layer configured to operate on a combination of the input image and the reference image.
The system of aspect 15, wherein:
- the image encoder and the text encoder are components of a multimodal encoder.
The system of aspect 15, wherein:
- the image generation model uses the image embedding and the text embedding for classifier-free guidance.
The system of aspect 15, further comprising:
- a user interface configured to obtain a selection input from a user, wherein a mask is created based on the selection input.
Dependencies are inferred from the claim text.