# US 12,602,851 B1 · Systems for incorporating objects into background images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,602,851 B1
- Document ID: US12602851B1
- Status: Granted
- Kind: B1
- Application number: 18/538,711
- Filed: 2023-12-13
- Granted: 2026-04-14
- Earliest claimed: 2023-12-13
- Assignees: Amazon Technologies, Inc.
- Applicants: Amazon Technologies, Inc.
- Inventors: Daiqing Qi; Shwetha Ram; Han Ding; Rui Meng; Changyou Chen; Tal Neiman
- Web page: https://patentgrep.com/patent/US12602851B1

## CPC

- G06T 11/60: Creating or editing images; Combining images with text
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- H04N 5/272: Means for inserting a foreground image in a background image, i.e. inlay, outlay
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A diffusion model receives as inputs an image of a background, first text describing the background, and second text describing an object to be added to the background. Using these inputs, the diffusion model generates an output image depicting the object within the background. To improve fidelity to the background image, the diffusion model uses the inputs associated with the background and the input associated with the object as separate conditions and determines a weight parameter that controls the probability of characteristics of each input occurring in the output image. Noise is added to the background image to create an initial image for the diffusion model. The first text and the second text are both used, in conjunction with the weight parameter, to determine a prompt to cause the diffusion model to generate an image of the object included in the background, while retaining the characteristics of the background image.

## Claims

### Claim 1

A system comprising:
one or more non-transitory memories storing computer-executable instructions; and
one or more hardware processors to execute the computer-executable instructions to:
receive user input associated with first image data;
determine one or more characteristics associated with the first image data;
determine first text data based on the one or more characteristics associated with the first image data, wherein the first image data represents first content;
determine second text data that represents second content;
provide the first text data and the second text data as inputs to a machine learning model that is trained to generate image data based on text data associated with one or more inputs that are dependent;
determine, using the machine learning model, second image data based on noise data and one or more of the first text data or the first image data;
determine, using the machine learning model, a weight parameter associated with one or more of the first text data or the second text data; and
determine, using the machine learning model, third image data based on the second image data, the weight parameter, the first text data, the second text data, and a parameter indicative of dependence between the first text data and the second text data.

### Claim 2 (depends on 1)

The system of claim 1, further comprising computer-executable instructions to:
determine the second text data based on the user input.

### Claim 3 (depends on 1)

The system of claim 1, further comprising computer-executable instructions to:
determine third text data based on the weight parameter, the first text data, and the second text data;
wherein the machine learning model determines the third image data based on the third text data.

### Claim 4 (depends on 1)

The system of claim 1, wherein the computer-executable instructions to determine the weight parameter include computer-executable instructions to, using the machine learning model:
determine an initial value for the weight parameter;
determine fourth image data based on the initial value, the first text data, and the second text data;
determine a loss value based on the fourth image data and one or more of the first image data or the first text data; and
modify the initial value based on the loss value to determine the weight parameter.

### Claim 5 (depends on 1)

The system of claim 1, wherein the machine learning model is a diffusion model that generates the second image data by adding noise to the first image data and generates the third image data by denoising the second image data based on the weight parameter, the first text data, and the second text data.

### Claim 6 (depends on 5)

The system of claim 5, wherein based on the weight parameter, the diffusion model determines:
a first probability of inclusion of first characteristics associated with the first text data in the third image data; and
a second probability of inclusion of second characteristics associated with the second text data in the third image data.

### Claim 7 (depends on 6)

The system of claim 6, wherein the weight parameter (λ) associated with the first text data (cs) and the second text data (ct) is used to determine the third image data based on the equation:
λct+(1−λ)cs.

### Claim 8 (depends on 1)

The system of claim 1, wherein the first content includes a background depicted in the first image data, the second content includes an object described by the second text data, and the third image data represents an image that depicts the object within the background.

### Claim 9

A system comprising:
one or more non-transitory memories storing computer-executable instructions; and
one or more hardware processors to execute the computer-executable instructions to:
determine first data associated with first content;
determine second data associated with second content;
provide the first data and the second data as inputs to a diffusion model that is trained to generate image data based on data associated with content;
determine, using the diffusion model, first image data based on the first data and noise data;
determine, using the diffusion model, a weight parameter associated with one or more of the first data or the second data;
determine third data based on the weight parameter, the first data, and the second data, and
determine, using the diffusion model, second image data based on the first image data, the third data, and a parameter indicative of dependence between the first data and the second data.

### Claim 10 (depends on 9)

The system of claim 9, further comprising computer-executable instructions to:
receive user input comprising an image;
determine one or more characteristics of the image; and
determine the first data based on the one or more characteristics.

### Claim 11 (depends on 9)

The system of claim 9, further comprising computer-executable instructions to:
receive user input comprising the first data, the second data, and an image, wherein the first data comprises text associated with the image and the second data comprises text associated with an object that is not included in the image.

### Claim 12 (depends on 9)

The system of claim 9, wherein the
third data comprises text data, and wherein the diffusion model determines the second image data based at least in part on the text data.

### Claim 13 (depends on 9)

The system of claim 9, wherein the computer-executable instructions to determine the weight parameter include computer-executable instructions to, using the diffusion model:
determine an initial value for the weight parameter;
determine third image data based on the initial value, the first data, and the second data;
determine a loss value based on the third image data and the first data; and
modify the initial value based on the loss value to determine the weight parameter.

### Claim 14 (depends on 9)

The system of claim 9, wherein the first content includes a first image that depicts a background, the second content includes text that describes an object, and the second image data represents a second image that depicts the object within the background.

### Claim 15 (depends on 9)

The system of claim 9, wherein based on the weight parameter, the diffusion model determines:
a first probability of inclusion of first characteristics associated with the first data in the second image data; and
a second probability of inclusion of second characteristics associated with the second data in the second image data.

### Claim 16 (depends on 15)

The system of claim 15, wherein the weight parameter (λ) associated with the first data (cs) and the second data (ct) is used to determine the second image data based on the equation:
λct+(1−λ)cs.

### Claim 17

A system comprising:
one or more non-transitory memories storing computer-executable instructions; and
one or more hardware processors to execute the computer-executable instructions to:
determine first data associated with first content;
determine second data associated with second content;
provide the first data and the second data as inputs to a machine learning model that is trained to generate image data based on data associated with content;
determine, using the machine learning model, first image data based on the first data and noise data;
determine, using the machine learning model, a weight parameter associated with one or more of the first data or the second data; and
determine, using the machine learning model, second image data based on the first image data, the weight parameter, the first data, the second data, and a parameter indicative of dependence between the first data and the second data, wherein based on the weight parameter, the machine learning model determines:
a first probability of inclusion of a first characteristic associated with the first data in the second image data; and
a second probability of inclusion of a second characteristic associated with the second data in the second image data.

### Claim 18 (depends on 17)

The system of claim 17, further comprising computer-executable instructions to determine the weight parameter by:
determining an initial value for the weight parameter;
determining third image data based on the initial value, the first data, and the second data;
determining a loss value based on the third image data and the first data; and
modifying the initial value based on the loss value to determine the weight parameter.

### Claim 19 (depends on 17)

The system of claim 17, further comprising computer-executable instructions to:
determine third data based on the weight parameter, the first data, and the second data, wherein the third data is used as a single input to the diffusion model.

### Claim 20 (depends on 17)

The system of claim 17, further comprising computer-executable instructions to:
receive user input comprising an image;
determine one or more characteristics of the image; and
determine the first data based on the one or more characteristics.