# US 12,008,739 B2 · Automatic photo editing via linguistic request

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,008,739 B2
- Document ID: US12008739B2
- Status: Granted
- Kind: B2
- Application number: 17/452,529
- Filed: 2021-10-27
- Published: 2023-04-27
- Granted: 2024-06-11
- Earliest claimed: 2021-10-27
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Ning Xu; Zhe Lin; Franck Dernoncourt
- Web page: https://patentgrep.com/patent/US12008739B2

## CPC

- G06T 5/77: Retouching; Inpainting; Scratch removal
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06T 5/50: using two or more images, e.g. averaging or subtraction
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/90: Dynamic range modification of images or parts thereof
- G06T 11/60: Creating or editing images; Combining images with text
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G10L 2015/223: {Execution procedure of a spoken command}
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

The present disclosure relates to systems and methods for automatically processing images based on a user request. In some examples, a request is divided into a retouching command (e.g., a global edit) and an inpainting command (e.g., a local edit). A retouching mask and an inpainting mask are generated to indicate areas where the edits will be applied. A photo-request attention and a multi-modal modulation process are applied to features representing the image, and a modified image that incorporates the user's request is generated using the modified features.

## Claims

### Claim 1

A method of image processing, comprising:
identifying an image and an edit command for the image;
encoding the edit command to obtain an inpainting vector and a retouching vector, wherein the inpainting vector indicates a local edit of the image and the retouching vector indicates a global edit of the image;
generating an inpainting mask based on the inpainting vector and a retouching mask based on the retouching vector;
generating an image feature representation based on the image and the inpainting mask;
generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix calculated using the retouching vector; and
generating a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the image.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
receiving the edit command from a user via an audio input device; and
transcribing the edit command, wherein the encoding is based on the transcribed edit command.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
generating a word embedding for each of a plurality of words of the edit command;
generating a probability vector that includes an inpainting probability and a retouching probability for each of the plurality of words;
applying the inpainting probability to a corresponding word of the plurality of words to obtain an inpainting weighted word representation, wherein the inpainting vector includes the inpainting weighted word representation; and
applying the retouching probability to the corresponding word of the plurality of words to obtain a retouching weighted word representation, wherein the retouching vector includes the retouching weighted word representation.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
encoding the image to generate image features;
identifying an inpainting object based on the edit command;
generating attention weights based on the edit command in the inpainting object; and
generating the inpainting mask based on the image features and the attention weights.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining whether the retouching vector indicates the global edit; and
generating the retouching mask based on the determination.

### Claim 6 (depends on 1)

The method of claim 1, further comprising:
encoding the image to generate image features;
applying the inpainting mask to the image features to obtain an inpainting image features; and
performing a convolution operation on the inpainting image features to obtain the image feature representation.

### Claim 7 (depends on 1)

The method of claim 1, further comprising:
calculating the attention matrix based on the image feature representation and the retouching vector;
expanding the retouching vector based on the retouching mask to obtain an expanded retouching vector; and
weighting the expanded retouching vector based on the attention matrix to obtain a weighted retouching vector.

### Claim 8 (depends on 7)

The method of claim 7, further comprising:
generating a scaling parameter and a shifting parameter based on the weighted retouching vector; and
adding the shifting parameter to a product of the scaling parameter and the image feature representation to obtain the modified image feature representation.

### Claim 9 (depends on 1)

The method of claim 1, further comprising:
performing a convolution operation on the modified image feature representation to obtain the modified image.

### Claim 10 (depends on 1)

The method of claim 1, wherein:
the inpainting vector indicates a local edit of the image.

### Claim 11 (depends on 1)

The method of claim 1, wherein:
the retouching vector indicates a global edit of the image.

### Claim 12

A method of training a neural network for image processing, comprising:
receiving training data comprising a training image, an edit command, and a ground truth image representing an application of the edit command to the training image;
encoding the edit command to obtain an inpainting vector and a retouching vector wherein the inpainting vector indicates a local edit of the training image and the retouching vector indicates a global edit of the training image;
generating an inpainting mask based on the inpainting vector and a retouching mask based on the retouching vector;
generating an image feature representation based on the training image and the inpainting mask;
generating a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix calculated using the retouching vector; and
generating a modified image based on the modified image feature representation, wherein the modified image represents the application of the edit command to the training image;
calculating a loss function based on the modified image and the ground truth image; and
training the neural network based on the loss function.

### Claim 13 (depends on 12)

The method of claim 12, further comprising:
computing an unconditional adversarial loss using a discriminator network based on the modified image and the ground truth image; and
training the neural network based on the unconditional adversarial loss.

### Claim 14 (depends on 12)

The method of claim 12, wherein:
the loss function comprises an L1 loss.

### Claim 15

An apparatus for image processing, comprising:
an encoder configured to encode an edit command for an image to obtain an inpainting vector and a retouching vector, wherein the inpainting vector indicates a local edit of the image and the retouching vector indicates a global edit of the image;
a phrase conditioned grounding (PCG) network configured to generate an inpainting mask based on the inpainting vector and a retouching mask based on the retouching vector;
a convolutional network configured to generate an image feature representation based on the image and the inpainting mask;
a multi-modal modulation network configured to generate a modified image feature representation based on the image feature representation, the retouching mask, and an attention matrix calculated using the retouching vector; and
a decoder configured to generate a modified image based on the modified image feature representation, wherein the modified image represents an application of the edit command to the image.

### Claim 16 (depends on 15)

The apparatus of claim 15, further comprising:
a photo request attention (PRA) network configured to calculate the attention matrix based on the image feature representation and the retouching vector.

### Claim 17 (depends on 16)

The apparatus of claim 16, wherein:
the PRA network comprises an embedding network configured to embed the image feature representation and the retouching vector into a common embedding space.

### Claim 18 (depends on 15)

The apparatus of claim 15, further comprising:
an audio input device configured to receive the edit command from a user, and to transcribe the edit command, wherein the encoding is based on the transcribed edit command.

### Claim 19 (depends on 15)

The apparatus of claim 15, wherein:
the encoder comprises a Bi-LSTM.

### Claim 20 (depends on 15)

The apparatus of claim 15, wherein:
the convolutional network comprises a gated convolution layer.