# US 2022/0036127 A1 · Semantic image manipulation using visual-semantic joint embeddings

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036127 A1
- Document ID: US20220036127A1
- Status: Published application
- Kind: A1
- Application number: 16/943,511
- Filed: 2020-07-30
- Published: 2022-02-03
- Earliest claimed: 2020-07-30
- Assignees: Adobe Inc.
- Applicants: Adobe Inc.
- Inventors: Zhe Lin; Xihui Liu; Quan Hung Tran; Jianming Zhang; Handong Zhao
- Web page: https://patentgrep.com/patent/US20220036127A1

## CPC

- G06T 11/60: Creating or editing images; Combining images with text
- G06F 18/214: Generating training patterns; Bootstrap methods, e.g. bagging or boosting
- G06F 18/22: Matching criteria, e.g. proximity measures
- G06T 11/00: Two-dimensional [2D] image generation
- G06V 10/44: Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
- G06V 10/774: Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
- G06V 10/82: using neural networks
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

The technology described herein is directed to a reinforcement learning based framework for training a natural media agent to learn a rendering policy without human supervision or labeled datasets. The reinforcement learning based framework feeds the natural media agent a training dataset to implicitly learn the rendering policy by exploring a canvas and minimizing a loss function. Once trained, the natural media agent can be applied to any reference image to generate a series (or sequence) of continuous-valued primitive graphic actions, e.g., sequence of painting strokes, that when rendered by a synthetic rendering environment on a canvas, reproduce an identical or transformed version of the reference image subject to limitations of an action space and the learned rendering policy.

## Claims

### Claim 1

One or more non-transitory computer readable storage media having a semantic image manipulation agent stored thereon, the semantic image manipulation agent including program instructions that, when executed by one or more processors, direct the one or more processors to:
encode a language-based editing instruction into textual feature vectors,
wherein the language-based editing instruction identifies a modification to a visual attribute of an input image;
embed the textual feature vectors into a visual-semantic joint embedding space;
manipulate an image feature map corresponding to the input image based on the textual feature vectors to modify the visual attribute of the input image within the visual-semantic joint embedding space; and
construct a new image from the manipulated image feature map.

### Claim 2 (depends on 1)

The one or more non-transitory computer readable storage media of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the one or more processors to:
extract edge information from the input image.

### Claim 3 (depends on 2)

The one or more non-transitory computer readable storage media of claim 2, wherein to construct the new image, the program instructions, when executed by the one or more processors, further direct the one or more processors to:
decode, using an image decoder, the new image based on the manipulated image feature map and the extracted edge information.

### Claim 4 (depends on 1)

The one or more non-transitory computer readable storage media of claim 1, wherein the program instructions, when executed by the one or more processors, further direct the one or more processors to:
encode the input image into the image feature map; and
embed the image feature map into the visual-semantic joint embedding space.

### Claim 5 (depends on 1)

The one or more non-transitory computer readable storage media of claim 1, wherein the program instructions, when executed by the one or more processors, pre-train a visual-semantic joint embedding model for semantic-based image manipulation by further directing the one or more processors to:
feed an image-caption pair to the visual-semantic joint embedding model,
wherein the image-caption pair includes a training image and a corresponding caption describing the training image;
encode the training image into a training image feature map;
perform an average pooling operation on the training image feature map to generate a training visual feature vector;
encode the caption describing the training image into a training textual feature vector;
normalize the training visual feature vector and the training textual feature vector;
calculate a matching score that indicates a similarity between the normalized training visual feature vector and the normalized training textual feature vector; and
iteratively train, utilizing a supervised learning operation, the visual-semantic joint embedding space based on the matching score.

### Claim 6 (depends on 5)

The one or more non-transitory computer readable storage media of claim 5, wherein the program instructions, when executed by the one or more processors, train an image decoder by further directing the one or more processors to:
feed a second training image to the pre-trained visual-semantic joint embedding model;
encode the second training image into a second training image feature map;
embed the second training image feature map into the visual-semantic joint embedding space;
reconstruct, using the image decoder, the second training image based on the second training image feature map;
determine a loss by comparing the second training image to the reconstructed second training image to determine a loss; and
feed the loss to the image decoder for training.

### Claim 7 (depends on 6)

The one or more non-transitory computer readable storage media of claim 6, wherein to determine the loss, the program instructions, when executed by the one or more processors, further direct the one or more processors to:
calculate a weighted sum of GAN loss, perceptual loss, and discriminator feature matching loss.

### Claim 8 (depends on 1)

The one or more non-transitory computer readable storage media of claim 1, wherein the textual feature vectors comprise:
a first textual feature vector corresponding to the visual attribute, and
a second textual feature vector corresponding to the modification to the visual attribute.

### Claim 9 (depends on 1)

The one or more non-transitory computer readable storage media of claim 1, wherein to manipulate the image feature map, the program instructions, when executed by the one or more processors, further direct the one or more processors to:
generate new feature map using a manipulation rule to replace the visual attribute of the input image identified by the first textual feature vector with the modification to the visual attribute identified by the second textual feature vector.

### Claim 10

A method of performing unsupervised open-vocabulary semantic image manipulation, the method comprising:
receiving an input image to be manipulated and a language-based editing instruction identifying a modification to a visual attribute of the input image;
encoding, using an image encoder, the input image into an image feature map;
encoding, using a text encoder, a language-based editing instruction into textual feature vectors including:
a first textual feature vector corresponding to the visual attribute, and
a second textual feature vector corresponding to the modification to the visual attribute;
embedding the image feature map and the textual feature vectors into a visual-semantic joint embedding space;
generating a new image feature map that modifies the visual attribute of the input image by manipulating the image feature map by the textual feature vectors within the visual-semantic joint embedding space; and
constructing, using an image decoder, a new image from the manipulated image feature map.

### Claim 11 (depends on 10)

The method of claim 10, further comprising:
extracting, using an edge extractor, edge information from the input image; and
feeding the edge information to the image decoder,
wherein constructing the new image includes decoding the new image from the new image feature map and the extracted edge information.

### Claim 12 (depends on 10)

The method of claim 10, further comprising pre-training the visual-semantic joint embedding space for semantic-based image manipulation by:
feeding an image-caption pair to the visual-semantic joint embedding model,
wherein the image-caption pair includes a training image and a corresponding caption describing the training image;
encoding, using the image encoder, the training image into a training image feature map;
performing an average pooling operation on the training image feature map to generate a training visual feature vector;
encoding, using the text encoder, the caption describing the training image into a training textual feature vector;
normalizing the training visual feature vector and the training textual feature vector;
calculating a matching score that indicates a similarity between the normalized training visual feature vector and the normalized training textual feature vector; and
using the matching score to iteratively train the visual-semantic joint embedding space.

### Claim 13 (depends on 12)

The method of claim 12, further comprising training an image decoder by:
encoding a second training image into a second training image feature map;
embedding the second training image feature map into the visual-semantic joint embedding space;
reconstructing, using the image decoder, the second training image based on the second training image feature map;
determining a loss by comparing the second training image to the reconstructed second training image to determine a loss; and
feeding the loss to the image decoder for training.

### Claim 14 (depends on 13)

The method of claim 13, wherein determining the loss comprises:
calculating a weighted sum of GAN loss, perceptual loss, and discriminator feature matching loss.

### Claim 15 (depends on 1)

The method of claim 1, wherein manipulating the image feature map by the features vectors within the visual-semantic joint embedding space comprises using a manipulation rule to replace the visual attribute of the input image identified by the first textual feature vector with the modification to the visual attribute identified by the second textual feature vector.

### Claim 16

A semantic image manipulation system, the system comprising:
a joint embedding model including:
an image encoder configured to encode an input image into an image feature map;
an image projection component configured to embed the image feature map into a visual-semantic joint embedding space;
a text encoder configured to encode a language-based editing instruction into textual feature vectors, wherein the language-based editing instruction identifies a modification to a visual attribute of the input image;
a text projection component configured to embed the textual feature vectors into a visual-semantic joint embedding space;
one or more processors configured to manipulate the image feature map corresponding to the input image based on the textual feature vectors to modify the visual attribute of the input image within the visual-semantic joint embedding space; and
an image decoder configured to decode the image feature map in the visual-semantic embedding space into a new image.

### Claim 17 (depends on 16)

The semantic-based image manipulation system of claim 16, further comprising:
an edge extractor configured to extract edge information from the input image, and
feed the edge information to the image decoder, wherein the image decoder is further configured to decode the new image based on the manipulated image feature map and the extracted edge information.

### Claim 18 (depends on 16)

The semantic-based image manipulation system of claim 16, wherein the edge extractor comprises a neural network-based edge extractor or Sobel edge extractor.

### Claim 19 (depends on 16)

The semantic-based image manipulation system of claim 16, wherein the language encoder comprises a long-short-term-memory network (LSTM).

### Claim 20 (depends on 16)

The semantic-based image manipulation system of claim 16, wherein the image encoder comprises a convolutional neural network.
