Generating digital images utilizing a diffusion-based network conditioned on lighting-aware feature representations

A system comprising:

• one or more memory devices;
• and
• one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
• generating, utilizing an environment lighting conditioning neural network, an environment lighting feature representation from an environment map corresponding to a target background image, wherein the environment lighting feature representation comprises a first encoding of one or more lighting features of the environment map;
• generating, utilizing a lighting conditioning neural network, a lighting feature representation from the target background image, wherein the lighting feature representation comprises a second encoding of one or more lighting features of the target background image;
• and
• modifying parameters of the lighting conditioning neural network by:
• determining an alignment loss based on differences between the lighting feature representation and the environment lighting feature representation in an encoding space;
• and
• updating the parameters of the lighting conditioning neural network to reduce the alignment loss by aligning the lighting feature representation with the environment lighting feature representation in the encoding space.

A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• determining, in response to a request to generate a digital image, a target background image for inserting a foreground object into the target background image;
• generating, from the target background image and utilizing a lighting conditioning neural network, a lighting feature representation comprising an encoding of one or more lighting features corresponding to one or more light sources of the target background image;
• and
• generating, utilizing a diffusion-based generative neural network conditioned on the lighting feature representation by injecting the lighting feature representation as conditional feature maps at a plurality of resolutions within the diffusion-based generative neural network, the digital image including the foreground object inserted into the target background image based on a composite image comprising the foreground object and the target background image, a foreground mask corresponding to the foreground object.