# US 2021/0233213 A1 · Attention-driven image manipulation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0233213 A1
- Document ID: US20210233213A1
- Status: Published application
- Kind: A1
- Application number: 16/752,030
- Filed: 2020-01-24
- Published: 2021-07-29
- Earliest claimed: 2020-01-24
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Youssef Alami Mejjati; Zoya Bylinskii; Elya Shechtman
- Web page: https://patentgrep.com/patent/US20210233213A1

## CPC

- G06T 7/11: Region-based segmentation
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 7/194: involving foreground-background segmentation
- G06V 10/26: Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06T 2200/24: involving graphical user interfaces [GUIs]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20092: Interactive image processing based on input by user
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Techniques of adjusting the salience of an image include generating values of photographic development parameters for a foreground and background of an image to adjust the salience of the image in the foreground. These parameters are global in nature over the image rather than local. Moreover, the optimization of the salience over such sets of global parameters is provided through two sets of these parameters by an encoder: one set corresponding to the foreground, in which the salience is to be either increased or decreased, and the other set corresponding to the background. Once the set of development parameters corresponding to the foreground region and the set of development parameters corresponding to the background region have been determined, a decoder generates an adjusted image with an increased salience based on these sets of development parameters.

## Claims

### Claim 1

A method, comprising:
receiving (i) initial image data representing an initial image having a plurality of pixels and (ii) object mask data, the object mask data indicating a foreground region of the plurality of pixels, the object mask data further indicating a background region of the plurality of pixels;
generating (i) foreground image parameter values corresponding to the foreground region and (ii) background image parameter values corresponding to the background region, the foreground image parameter values and the background image parameter values being based on the initial image data and the object mask data;
generating adjusted image data representing an adjusted image, the adjusted image being based on the initial image data, the foreground image parameter values, and the background image parameter values, the adjusted image having different salience values in the foreground region than the initial image; and
outputting the adjusted image.

### Claim 2 (depends on 1)

The method as in claim 1,
wherein generating the foreground image parameter values and the background image parameter values includes inputting the initial image data and the object mask data into an encoder of a machine learning engine, the encoder including a plurality of neural networks, a first neural network of the plurality of neural networks outputting the foreground image parameter values and the background image parameter values. and
wherein generating the adjusted image data includes inputting the initial image data, the foreground image parameter values, and the background image parameter values into a decoder of the machine learning engine, the decoder including a fixed set of functions, each of the fixed set of functions being based on the foreground image parameter values and the background image parameter values and being applied to the initial image data to produce the adjusted image values.

### Claim 3 (depends on 2)

The method as in claim 2, wherein generating the foreground image parameter values and the background image parameter values further includes:
inputting the initial image data into a second neural network of the plurality of neural networks to produce a first output indicative of semantic content of the initial image;
generating initial salience data representing a saliency map of the initial image;
inputting the object mask data and the initial salience data into a third neural network of the plurality of neural networks to produce a second output indicative of a difference between the saliency map of the foreground region of the initial image and a saliency map of the foreground region of the adjusted image;
combining the first output and the second output to produce a third output; and
inputting the third output into the neural network of the plurality of neural networks to produce the foreground image parameter values and the background image parameter values upon being input.

### Claim 4 (depends on 3)

The method as in claim 3, wherein the first output includes a vector of a specified size, each element of the vector including a representation of a feature extracted from the initial image data.

### Claim 5 (depends on 4)

The method as in claim 4, wherein the representation of a feature included in an element of the vector has a size based on a layer of the second neural network from which the representation of the feature is extracted.

### Claim 6 (depends on 3)

The method as in claim 3, further comprising performing a global average pooling operation on the first output.

### Claim 7 (depends on 3)

The method as in claim 3, further comprising performing a global average pooling operation on the second output.

### Claim 8

A system configured to adjust a salience of selected regions of images, the system comprising:
at least one memory including instructions; and
at least one processor that is operably coupled to the at least one memory and that is arranged and configured to execute instructions that, when executed, cause the at least one processor to implement an application, the application comprising:
a graphical user interface (GUI) configured to indicate (i) initial image data representing an initial image having a plurality of pixels and (ii) object mask data indicating a foreground region of the plurality of pixels and a background region of the plurality of pixels, the GUI being displayed in a display device;
a machine learning engine including:
an encoder configured to generate (i) foreground image parameter values corresponding to the foreground region and (ii) background image parameter values corresponding to the background region based on the initial image data and the object mask data;
a decoder configured to generate adjusted image data representing an adjusted image, the adjusted image being based on the initial image data, the foreground image parameter values, and the background image parameter values, the adjusted image having increased salience values in the foreground region; and
a display engine configured to display the adjusted image in an output window of the GUI.

### Claim 9 (depends on 8)

The system as in claim 8, wherein the adjusted image data is produced from the initial image data and the object mask data based on a machine learning engine, the machine learning engine including an encoder and a decoder,
the encoder including a plurality of neural networks, a first neural network of the plurality of neural networks outputting the foreground image parameter values and the background image parameter values,
the decoder including a fixed set of functions, each of the fixed set of functions being based on the foreground image parameter values and the background image parameter values and being applied to the initial image data to produce the adjusted image values,
wherein the instructions that, when executed, cause the at least one processor to generate the foreground image parameter values and the background image parameter values further cause the at least one processor to input the initial image data and the object mask data into the encoder. and
wherein the instructions that, when executed, cause the at least one processor to generate the adjusted image data further cause the at least one processor to input the initial image data, the foreground image parameter values, and the background image parameter values into the decoder.

### Claim 10 (depends on 9)

The system as in claim 9, wherein the first neural network includes fully connected layers.

### Claim 11 (depends on 9)

The system as in claim 9, wherein the instructions that, when executed, cause the at least one processor to generate the adjusted image data further cause the at least one processor to:
generate an intermediate foreground image based on (i) the initial image in the foreground region and the foreground image parameter values and (ii) the initial image in the background region and the background image parameter values; and
generate an intermediate background image based on (i) the initial image in the foreground region and the foreground image parameter values and (ii) the initial image in the background region and the background image parameter values.

### Claim 12 (depends on 11)

The system as in claim 11, wherein the instructions that, when executed, cause the at least one processor to generate the intermediate foreground image further cause the at least one processor to:
apply the decoder to the initial image in the foreground region and the foreground image parameter values to produce a first output;
apply the decoder to the initial image in the background region and the background image parameter values to produce a second output; and
produce, as the intermediate foreground image, a sum of (i) a Hadamard product of the first output and the object mask data and (ii) a Hadamard product of the second output and a complement of the object mask data.

### Claim 13 (depends on 11)

The system as in claim 11, wherein the instructions that, when executed, cause the at least one processor to generate the intermediate background image further cause the at least one processor to:
apply the decoder to the initial image in the foreground region and the foreground image parameter values to produce a first output;
apply the decoder to the initial image in the background region and the background image parameter values to produce a second output; and
produce, as the intermediate background image, a sum of (i) a Hadamard product of the first output and the object mask data and (ii) a Hadamard product of the second output and a complement of the object mask data.

### Claim 14 (depends on 11)

The system as in claim 11, wherein the intermediate foreground image is a first intermediate foreground image and the intermediate background image is a first intermediate background image. and
wherein the adjusted image is a sum of (i) a Hadamard product of a second intermediate foreground image and the object mask data and (ii) a Hadamard product of a second intermediate background image and a complement of the object mask data, each of the second intermediate foreground image and the second intermediate background image being based on the first intermediate foreground image, the first intermediate background image, the foreground image parameter values, and the background image parameter values.

### Claim 15

A computer program product including a non-transitory computer-readable storage medium and storing executable code that, when executed by at least one data processing apparatus configured to adjust a salience of selected regions of images, is configured to cause the at least one data processing apparatus to perform a method, the method comprising:
receiving (i) initial image data representing an initial image having a plurality of pixels and (ii) object mask data indicating a foreground region of the plurality of pixels and a background region of the plurality of pixels;
generating (i) foreground image parameter values corresponding to the foreground region and (ii) background image parameter values corresponding to the background region, the foreground image parameter values and the background image parameter values being based on the initial image data and the object mask data;
generating adjusted image data representing an adjusted image, the adjusted image being based on the initial image data, the foreground image parameter values, and the background image parameter values, the adjusted image having increased salience values in the foreground region; and
displaying the adjusted image on a display device.

### Claim 16 (depends on 15)

The computer program product as in claim 15, wherein the adjusted image data is produced from the initial image data and the object mask data based on a machine learning engine, the machine learning engine including an encoder and a decoder,
the encoder including a plurality of neural networks, a first neural network of the plurality of neural networks outputting the foreground image parameter values and the background image parameter values,
the decoder including a fixed set of functions, each of the fixed set of functions being applied separately to the foreground image parameter values and the background image parameter values to produce the adjusted image values,
wherein generating the foreground image parameter values and the background image parameter values includes inputting the initial image data and the object mask data into the encoder. and
wherein generating the adjusted image data includes inputting the initial image data, the foreground image parameter values, and the background image parameter values into the decoder.

### Claim 17 (depends on 16)

The computer program product as in claim 16, wherein the machine learning engine includes a loss function, the loss function having a salience component and an adversarial component, the salience component of the cost function representing a difference between the salience values of the foreground region of the initial image and salience values of the foreground region of the adjusted image, the adversarial component representing a likelihood that the adjusted image data is recognizable as a real image to a viewer.

### Claim 18 (depends on 17)

The computer program product as in claim 17, wherein the loss function further has a perceptual loss component, the perceptual loss component being based on a difference between feature representations extracted from the initial image and the adjusted image.

### Claim 19 (depends on 16)

The computer program product as in claim 16, wherein the encoder further includes a control configured to allow a user to enable or disable a parameter of the foreground image parameters and/or the background image parameters.

### Claim 20 (depends on 16)

The computer program product as in claim 16, wherein generating the foreground image parameter values and the background image parameter values further includes:
inputting the initial image data into a second neural network of the plurality of neural networks to produce a first output indicative of semantic content of the initial image; and
wherein the encoder further includes a multi-layer perceptron configured to modulate the first output with a randomly sampled vector having affine transformation parameters, values of the affine transformation parameters indicating a personal style of a user.
