Generating physically-based rendering (PBR) textures using image-based diffusion

A non-transitory computer-readable storage medium storing a set of instructions that, when executed by one or more computer processors, causes the one or more computer processors to perform operations, the operations comprising:

• receiving a 3D mesh input;
• receiving one or more text prompts associated with the 3D mesh input;
• inputting the 3D mesh input and the one or more text prompts into a trained machine-learning model, wherein the trained machine-learning model includes a cross-domain attention mechanism that aligns the feature spaces of a plurality of parallel networks, the plurality of parallel networks including a red, green, blue (RGB) space trained on RGB images and a physically-based rendering (PBR) trained on PBR textures;
• and
• receiving output data from the trained machine-learning model, the output data comprising material and texture information, wherein the material and texture information enables rendered views of a 3D mesh with the output data to respond to changes in lighting in a real-time 3D rendering environment.

A method comprising:

• receiving a 3D mesh input;
• receiving one or more text prompts associated with the 3D mesh input;
• inputting the 3D mesh input and the one or more text prompts into a trained machine-learning model, wherein the trained machine-learning model includes a cross-domain attention mechanism that aligns the feature spaces of a plurality of parallel networks, the plurality of parallel networks including a red, green, blue (RGB) space trained on RGB images and a physically-based rendering (PBR) trained on PBR textures;
• and
• receiving output data from the trained machine-learning model, the output data comprising material and texture information, wherein the material and texture information enables rendered views of a 3D mesh with the output data to respond to changes in lighting in a real-time 3D rendering environment.