Back to results
US 12,592,030 B2Granted
Interactive three-dimension aware text-to-image generation
AssigneeAdobe Inc.
InventorsMatheus Gadelha, Tomasz Opasinski, Kevin James Blackburn-Matzen, Mathieu Kevin Pascal Gaillard, Giorgio Gori, Radomir Mech
Life of this patent
Estimating the term…
Abstract
An image processing system is configured to receive a three-dimensional (3D) model and a text prompt that describes a scene corresponding to the 3D model. The system may then generate a depth map of the 3D model and generate an output image based on the depth map and the text prompt. The output image may depicts a view of the scene that includes textures described by the text prompt. The output image may be generated using an image generation model.
Claims16
A method comprising:
- receiving, via a combined interface of a three-dimensional (3D) modeling application and an image generation model, a 3D edit input and a text prompt, wherein the 3D edit input indicates a change to a 3D model depicting an object, the text prompt describes a scene corresponding to the 3D model, and the text prompt describes a texture for the object, and wherein the combined interface of the 3D modeling application and the image generation model comprises a modeling interface for editing the 3D model, a text interface for receiving the texture, and a preview element for displaying an output of the image generation model;
- modifying, using the 3D modeling application, the 3D model based on the 3D edit input to obtain a modified 3D model;
- generating a depth map of the modified 3D model;
- generating, by the image generation model, an output image by denoising input noise based on the depth map and the text prompt to obtain image features representing the scene and decoding the image features to obtain the output image, wherein the output image depicts a view of the scene including the object with the texture;
- and
- displaying the output image via the preview element of the combined interface.
3 more elements
An apparatus comprising:
- at least one processor;
- at least one memory storing instructions and in electronic communication with the at least one processor;
- a combined interface for a three-dimensional (3D) modeling application and an image generation model, wherein the combined interface is configured to receive a 3D edit input and a text prompt, wherein the 3D edit input indicates a change to a 3D model depicting an object, the text prompt describes a scene corresponding to the 3D model, and the text prompt describes a texture for the object, wherein the combined interface of the 3D modeling application and the image generation model comprises a modeling interface for editing the 3D model, a text interface for receiving the texture, and a preview element for displaying an output of the image generation model;
- the 3D modeling application configured to generate a depth map by modifying the 3D model;
- and
- the image generation model configured to generate an output image by denoising input noise based on the depth map and a text prompt to obtain image features representing the scene and decoding the image features to obtain the output image, wherein the output image depicts a view of a scene including the object with the texture.
3 more elements
A non-transitory computer readable medium storing code for image processing, the code comprising instructions executable by a processor to:
- receive, via a combined interface of a three-dimensional (3D) modeling application and an image generation model, a 3D edit input and a text prompt, wherein the 3D edit input indicates a change to a 3D model depicting an object, the text prompt describes a scene corresponding to the 3D model, and the text prompt describes a texture for the object, and wherein the combined interface of the 3D modeling application and the image generation model comprises a modeling interface for editing the 3D model, a text interface for receiving the texture, and a preview element for displaying an output of the image generation model;
- modify, using the 3D modeling application, the 3D model based on the 3D edit input to obtain a modified 3D model;
- generate a depth map of the modified 3D model;
- generate, using the image generation model, an output image by denoising input noise based on the depth map and the text prompt to obtain image features representing the scene and decoding the image features to obtain the output image, wherein the output image depicts a view of the scene including the object with the texture;
- and
- display the output image via the preview element of the combined interface.
3 more elements
Dependencies are inferred from the claim text.