Prompt-driven image editing using machine learning

A non-transitory computer-readable medium with instructions stored thereon that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• receiving an initial image and a textual request to change the initial image, the initial image including a subject with a face;
• generating a preserving mask that corresponds to the face of the subject;
• providing the textual request, the initial image, and the preserving mask as input to a diffusion model;
• outputting, with the diffusion model, a denoised initial image based on the initial image;
• performing, with the diffusion model, text conditioning of the textual request and forward diffusion to generate a noisy translated image that satisfies the textual request;
• outputting, with the diffusion model, based on the noisy translated image, extracted features, and self-attention maps, a denoised translated image;
• and
• blending the denoised initial image, the preserving mask, and the denoised translated image to form an output image, wherein the preserving mask prevents modification to the face from the initial image.

A system comprising:

• a processor;
• and
• a memory coupled to the processor, with instructions stored thereon that, when executed by the processor, cause the processor to perform operations comprising:
• receiving an initial image and a textual request to change the initial image, the initial image including a subject with a face;
• generating a preserving mask that corresponds to the face of the subject;
• providing the textual request, the initial image, and the preserving mask as input to a diffusion model;
• outputting, with the diffusion model, a denoised initial image based on the initial image;
• performing, with the diffusion model, text conditioning of the textual request and forward diffusion to generate a noisy translated image that satisfies the textual request;
• outputting, with the diffusion model, based on the noisy translated image, extracted features, and self-attention maps, a denoised translated image;
• and
• blending the denoised initial image, the preserving mask, and the denoised translated image to form an output image, wherein the preserving mask prevents modification to the face from the initial image.