Image generation using a text and image conditioned machine learning model
AssigneeAdobe Inc.
InventorsTobias Hinz, Venkata Naveen Kumar Yadav Marri, Midhun Harikumar, Ajinkya Gorakhnath Kale, Zhe Lin, Oliver Wang, Jingwan Lu
Life of this patent
Estimating the term…
Abstract
A method, apparatus, non-transitory computer readable medium, and system for image generation include obtaining a text embedding of a text prompt and an image embedding of an image prompt. Some embodiments map the text embedding into a joint embedding space to obtain a joint text embedding and map the image embedding into the joint embedding space to obtain a joint image embedding. Some embodiments generate a synthetic image based on the joint text embedding and the joint image embedding.
Claims20
A method for image generation, comprising:
- obtaining a text embedding of a text prompt in a text embedding space and an image embedding of an image prompt in an image embedding space;
- mapping, using a text mapping network, the text embedding from the text embedding space into a joint embedding space to obtain a joint text embedding;
- mapping, using an image mapping network, the image embedding from the image embedding space into the joint embedding space to obtain a joint image embedding;
- and
- generating, using an image generation model, a synthetic image based on the joint text embedding and the joint image embedding.
A system for image generation, comprising:
- one or more processors;
- one or more memory components coupled with the one or more processors;
- a text mapping network comprising text mapping parameters, the text mapping network trained to map a text embedding from a text embedding space into a joint embedding space to obtain a joint text embedding;
- an image mapping network comprising image mapping parameters, the image mapping network trained to map an image embedding from an image embedding space into the joint embedding space to obtain a joint image embedding;
- and
- an image generation model comprising image generation parameters, the image generation model trained to generate a synthetic image based on the joint text embedding and the joint image embedding.
A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to:
- obtain a text embedding of a text prompt in a text embedding space and an image embedding of an image prompt in an image embedding space;
- map, using a text mapping network, the text embedding from the text embedding space into a joint embedding space to obtain a joint text embedding;
- map, using an image mapping network, the image embedding from the image embedding space into the joint embedding space to obtain a joint image embedding;
- and
- generate, using an image generation model, a synthetic image based on the joint text embedding and the joint image embedding.
Dependencies are inferred from the claim text.