# US 2025/0095254 A1 · Methods and systems for prompt-based image editing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0095254 A1
- Document ID: US20250095254A1
- Status: Published application
- Kind: A1
- Application number: 18/882,265
- Filed: 2024-09-11
- Published: 2025-03-20
- Earliest claimed: 2023-09-14
- Assignees: Canva Pty Ltd
- Applicants: Canva Pty Ltd
- Inventors: Stefan Paul SIETZEN; Sanchit SANCHIT; Alexander TACK
- Web page: https://patentgrep.com/patent/US20250095254A1

## CPC

- G06T 11/60: Creating or editing images; Combining images with text
- G06T 5/70: Denoising; Smoothing
- G06T 2210/22: Cropping
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Some embodiments relate to a method of performing prompt-based image editing. The method includes accessing an image; receiving a selected area of the image; receiving a prompt, wherein the prompt is indicative of an editing instruction; generating a latent by transforming the image into visual noise; predicting, based on the latent and the prompt, a noise image corresponding to the latent; subtracting at least a portion of the noise image from the latent to generate an updated latent; generating a noisy representation of the image; generating a masked latent based on the noisy representation, the updated latent and selected area, wherein the masked latent comprises the updated latent in the areas corresponding to the selected area, and the noisy representation in the areas that do not correspond to the selected area.

## Claims

### Claim 1

A method of performing prompt-based image editing, the method comprising:
accessing an image;
receiving a selected area of the image;
receiving a prompt, wherein the prompt is indicative of an editing instruction;
generating a latent by transforming the image into visual noise;
predicting, based on the latent and the prompt, a noise image corresponding to the latent;
subtracting at least a portion of the noise image from the latent to generate an updated latent;
generating a noisy representation of the image;
generating a masked latent based on the noisy representation, the updated latent and selected area, wherein the masked latent comprises the updated latent in the areas corresponding to the selected area, and the noisy representation in the areas that do not correspond to the selected area.

### Claim 2 (depends on 1)

The method of claim 1, further comprising repeating the steps of a denoising loop over a series of timesteps, wherein the denoising loop comprises the steps of predicting a noise image, generating an updated latent, generating a noisy representation and generating a masked latent.

### Claim 3 (depends on 2)

The method of claim 2, wherein given a current timestep t, predicting the noise image comprises predicting the visual noise that would be present in the latent at a timestep t+1 during a noising process.

### Claim 4 (depends on 2)

The method of claim 2, wherein generating the noisy representation of the first image comprises adding noise to the image based on the current timestep t.

### Claim 5 (depends on 2)

The method of claim 2, wherein the timestep t is decremented after each iteration.

### Claim 6 (depends on 1)

The method of claim 1, wherein the portion of the noise image to subtract from the latent is determined based on a noise schedule.

### Claim 7 (depends on 1)

The method of claim 1, wherein the step of predicting the noise image is performed by a diffusion model, wherein the diffusion model is trained using sets of data comprising a first image, an editing instruction, and a second image edited based on the editing instruction.

### Claim 8 (depends on 7)

The method of claim 7, wherein the diffusion model is trained by being caused to add noise to the first image with timesteps t increasing, then caused to remove noise by decrementing timesteps t to generate the second edited image

### Claim 9 (depends on 1)

The method of claim 1, further comprising determining an encoding of the received prompt, wherein the step of predicting a noise image is done using the encoding of the prompt.

### Claim 10 (depends on 1)

The method of claim 1, further comprising cropping the accessed image based on the selected area.

### Claim 11 (depends on 1)

The method of claim 1, further comprising altering the resolution of the accessed image to a working resolution before generating the latent, wherein the working resolution is a lower resolution that the resolution of the accessed image.

### Claim 12 (depends on 1)

The method of claim 1, further comprising mapping pixel information corresponding to the accessed image to a latent space before generating the latent, such that the latent is a representation of the image in the latent space.

### Claim 13 (depends on 1)

The method of claim 1, further comprising inserting the generated masked latent into the accessed image to produce an output image.

### Claim 14 (depends on 13)

The method of claim 13, further comprising performing a pixel melding process on the generated masked latent to blend the generated masked latent with the accessed image.

### Claim 15 (depends on 14)

The method of claim 14, further comprising outputting the image by saving the output image to a memory location, displaying the output image to a user, or sending the output image to an external device.

### Claim 16 (depends on 1)

The method of claim 1, further comprising generating a binary mask based on the selected area, and using the binary mask to generate the masked latent.

### Claim 17 (depends on 1)

The method of claim 1, wherein the masked latent is computed based on the equation
masked_latent
=
mask
*
latent
+
(
1
-
mask
)
*
noisy_representation
,
where mask represents a mask generated based on the selected area, latent represents the updated latent, and noisy_representation represents the noisy representation.

### Claim 18 (depends on 1)

The method of claim 1, further comprising receiving an indication that the prompt relates to a localised edit.

### Claim 19

A non-transitory computer-readable storage medium storing instructions which, when executed by a processing device, cause the processing device to perform a method, the method comprising:
accessing an image;
receiving a selected area of the image;
receiving a prompt, wherein the prompt is indicative of an editing instruction;
generating a latent by transforming the image into visual noise;
predicting, based on the latent and the prompt, a noise image corresponding to the latent;
subtracting at least a portion of the noise image from the latent to generate an updated latent;
generating a noisy representation of the image;
generating a masked latent based on the noisy representation, the updated latent and selected area, wherein the masked latent comprises the updated latent in the areas corresponding to the selected area, and the noisy representation in the areas that do not correspond to the selected area.

### Claim 20 (depends on 19)

A computing device comprising:
the non-transitory computer-readable storage medium of claim 19; and
a processor configured to execute the instructions stored in the non-transitory computer-readable storage medium.
