High-quality three dimensional asset generation

A method for generating a three-dimensional (3D) asset, the method comprising:

• receiving and processing, using a first multi-view diffusion model, an input prompt to generate a muti-view image of an object for which the 3D asset is to be generated;
• receiving and processing, using a second multi-view diffusion model, the input prompt and the multi-view image to generate a multi-view surface normal image of the object;
• receiving and processing, using a 3D reconstruction engine, the multi-view image and the multi-view surface normal image to generate an intermediate 3D representation of the object, wherein the intermediate 3D representation comprises a polygon mesh and a low-resolution texture map;
• rendering, using a rendering engine, a second multi-view image based on the intermediate 3D representation;
• receiving and processing, using a third multi-view diffusion model the input prompt, the rendered second multi-view image, and the low-resolution texture map to generate a high-resolution multi-view image;
• upscaling, using the 3D reconstruction engine, the low-resolution texture map based on the high-resolution multi-view image;
• and
• providing the 3D asset, the 3D asset comprising the polygon mesh and the upscaled texture map.

A non-transitory computer-readable medium having stored thereon instructions that, when executed by processing circuitry, cause the processing circuitry to perform a method for generating a three-dimensional (3D) asset, the method comprising:

• receiving and processing, using a first multi-view diffusion model, an input prompt to generate a muti-view image of an object for which the 3D asset is to be generated;
• receiving and processing, using a second multi-view diffusion model, the input prompt and the multi-view image to generate a multi-view surface normal image of the object;
• receiving and processing, using a 3D reconstruction engine, the multi-view image and the multi-view surface normal image to generate an intermediate 3D representation of the object, wherein the intermediate 3D representation comprises a polygon mesh and a low-resolution texture map;
• rendering, using a rendering engine, a second multi-view image based on the intermediate 3D representation;
• receiving and processing, using a third multi-view diffusion model the input prompt, the rendered second multi-view image, and the low-resolution texture map to generate a high-resolution multi-view image;
• upscaling using the 3D reconstruction engine, the low-resolution texture map based on the high-resolution multi-view image;
• and
• providing the 3D asset, the 3D asset comprising the polygon mesh and the upscaled texture map.

A system comprising:

• processing circuitry to generate a three-dimensional (3D) asset using one or more neural networks, the processing circuitry being configured to implement:
• a first multi-view diffusion model configured to receive and process an input prompt to generate a muti-view image of an object for which the 3D asset is to be generated;
• a second multi-view diffusion model configured to receive and process the input prompt and the multi-view image to generate a multi-view surface normal image of the object;
• a 3D reconstruction engine configured to receive and process the multi-view image and the multi-view surface normal image to generate an 3D representation of the object, wherein the 3D representation comprises a polygon mesh and a texture map;
• and
• one or more memories to store parameters associated with the one or more neural networks.

A system comprising:

• processing circuitry to generate a three-dimensional (3D) asset using one or more neural networks, the processing circuitry being configured to implement:
• a 3D reconstruction engine configured to receive and process a multi-view image and a multi-view surface normal image to generate an intermediate 3D representation, wherein the intermediate 3D representation comprises a polygon mesh and a low-resolution texture map;
• a rendering engine configured to render a second multi-view image based on the intermediate 3D representation of the object;
• and
• a third multi-view diffusion model configured to receive and process the input prompt and the rendered second multi-view image to generate a high-resolution multi-view image;
• and
• one or more memories to store parameters associated with the one or more neural networks,
• wherein the 3D asset comprises the polygon mesh and a high-resolution texture map generated by upscaling, by the 3D reconstruction engine, the low-resolution texture map based on the high-resolution multi-view image.