Generating portraits according to relighting schemes using diffusion models trained on synthetic image data

A system comprising:

• one or more memory devices;
• and
• one or more processors coupled to the one or more memory devices that cause the system to perform operations comprising:
• accessing a first synthetic digital image comprising a first portrait of a subject;
• accessing an environment map defining a new lighting scheme for the subject;
• generating, utilizing a diffusion model, a second synthetic digital image comprising a relit portrait of the subject based on the first synthetic digital image and the environment map;
• determining a measure of loss based on the second synthetic digital image and a ground truth synthetic digital image comprising a second portrait of the subject with altered lighting according to the new lighting scheme of the environment map;
• and
• adjusting parameters of the diffusion model to reduce differences between the second synthetic digital image and the ground truth synthetic digital image based on the measure of loss.

A non-transitory computer-readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:

• generating, utilizing a diffusion model, a first noised image based on a first digital image comprising a portrait of a subject and based on an environment map defining a new lighting scheme for the subject;
• generating, utilizing the diffusion model, a second noised image based on the environment map and a text prompt comprising a description of the subject;
• and
• generating, utilizing the diffusion model, a new digital image comprising a relit portrait of the subject by combining the first noised image and the second noised image.