# US 2020/0349391 A1 · Method for training image generation network, electronic device, and storage medium

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0349391 A1
- Document ID: US20200349391A1
- Status: Published application
- Kind: A1
- Application number: 16/857,337
- Filed: 2020-04-24
- Published: 2020-11-05
- Earliest claimed: 2019-04-30
- Applicants: Shenzhen Sensetime Technology Co., Ltd.
- Inventors: Yu ZHANG; Dongqing Zou; Sijie Ren; Zhe Jiang; Xiaohao Chen
- Web page: https://patentgrep.com/patent/US20200349391A1

## CPC

- G06N 3/08: Learning methods
- G06F 18/2148: {characterised by the process organisation or structure, e.g. boosting cascade}
- G06F 18/2178: {based on feedback of a supervisor}
- G06F 18/28: Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06V 10/764: using classification, e.g. of video objects
- G06V 10/82: using neural networks
- G06V 20/64: Three-dimensional [3D] objects
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for training an image generation network, an electronic device and a storage medium are provided. The method includes: obtaining a sample image, where the sample image includes a first sample image and a second sample image corresponding to the first sample image; processing the first sample image based on an image generation network to obtain a predicted target image; determining a difference loss between the predicted target image and the second sample image; and training the image generation network based on the difference loss to obtain a trained image generation network.

## Claims

### Claim 1

A method for training an image generation network, comprising:
obtaining a sample image, wherein the sample image comprises a first sample image and a second sample image corresponding to the first sample image;
processing the first sample image based on an image generation network to obtain a predicted target image;
determining a difference loss between the predicted target image and the second sample image; and
training the image generation network based on the difference loss to obtain a trained image generation network.

### Claim 2 (depends on 1)

The method according to claim 1, wherein determining the difference loss between the predicted target image and the second sample image comprises:
determining the difference loss between the predicted target image and the second sample image based on a structure analysis network; and
training the image generation network based on the difference loss to obtain the trained image generation network comprises:
performing adversarial training on the image generation network and the structure analysis network based on the difference loss to obtain the trained image generation network.

### Claim 3 (depends on 2)

The method according to claim 2, wherein the difference loss comprises a first structural difference loss and a feature loss;
determining the difference loss between the predicted target image and the second sample image comprises:
processing the predicted target image and the second sample image based on the structure analysis network to determine the first structural difference loss between the predicted target image and the second sample image; and
determining the feature loss between the predicted target image and the second sample image based on the structure analysis network.

### Claim 4 (depends on 3)

The method according to claim 3, wherein processing the predicted target image and the second sample image based on the structure analysis network to determine the first structural difference loss between the predicted target image and the second sample image comprises:
processing the predicted target image based on the structure analysis network to determine at least one first structural feature of at least one position in the predicted target image;
processing the second sample image based on the structure analysis network to determine at least one second structural feature of at least one position in the second sample image; and
determining the first structural difference loss between the predicted target image and the second sample image based on the at least one first structural feature and the at least one second structural feature.

### Claim 5 (depends on 4)

The method according to claim 4, wherein processing the predicted target image based on the structure analysis network to determine the at least one first structural feature of the at least one position in the predicted target image comprises:
processing the predicted target image based on the structure analysis network to obtain at least one first feature map in at least one scale of the predicted target image; and
obtaining, for each of the at least one first feature map, the at least one first structural feature of the predicted target image based on a cosine distance between a feature of each of at least one position in the first feature map and a feature of an adjacent region to the position, wherein each position in the first feature map corresponds to one first structural feature, and the feature of the adjacent region is each feature in a region centered on the position and comprising at least two positions.

### Claim 6 (depends on 4)

The method according to claim 4, wherein processing the second sample image based on the structure analysis network to determine the at least one second structural feature of the at least one position in the second sample image comprises:
processing the second sample image based on the structure analysis network to obtain at least one second feature map in at least one scale of the second sample image; and
obtaining, for each of the at least one second feature map, the at least one second structural feature of the second sample image based on a cosine distance between a feature of each of at least one position in the second feature map and a feature of an adjacent region to the position, wherein each position in the second feature map corresponds to one second structural feature.

### Claim 7 (depends on 6)

The method according to claim 6, wherein the each position in the first feature map has a correspondence with the each position in the second feature map;
wherein determining the first structural difference loss between the predicted target image and the second sample image based on the at least one first structural feature and the at least one second structural feature comprises:
calculating a distance between the first structural feature corresponding to a position in the first feature map and the second structural feature corresponding to a position in the second feature map having a correspondence to the position, in the first feature map; and
determining the first structural difference loss between the predicted target image and the second sample image based on distances between all first structural features and second structural features corresponding to the predicted target image.

### Claim 8 (depends on 3)

The method according to claim 3, wherein determining the feature loss between the predicted target image and the second sample image based on the structure analysis network comprises:
processing the predicted target image and the second sample image based on the structure analysis network to obtain the first feature map in the at least one scale of the predicted target image and the second feature map in the at least one scale of the second sample image; and
determining the feature loss between the predicted target image and the second sample image based on at least one first feature map and at least one second feature map.

### Claim 9 (depends on 8)

The method according to claim 8, wherein each position in the first feature map has a correspondence with each position in the second feature map;
determining the feature loss between the predicted target image and the second sample image based on the at least one first feature map and the at least one second feature map comprises:
calculating a distance between a feature in the first feature map and a feature in the second feature map respectively corresponding to the positions having a correspondence; and
determining the feature loss between the predicted target image and the second sample image based on the feature in the first feature map and the feature in the second feature map.

### Claim 10 (depends on 3)

The method according to claim 3, wherein the difference loss further comprises a color loss; and before training the image generation network based on the difference loss to obtain the trained image generation network, the method further comprises:
determining a color loss of the image generation network based on the color loss between the predicted target image and the second sample image;
wherein performing the adversarial training on the image generation network and the structure analysis network based on the difference loss to obtain the trained image generation network comprises:
adjusting, in a first iteration, a network parameter in the image generation network based on the first structural difference loss, the feature loss, and the color loss;
adjusting, in a second iteration, the network parameter in the structure analysis network based on the first structural difference loss, wherein the first iteration and the second iteration are two continuously-executed iterations; and
obtaining the trained image generation network when a training stopping condition is satisfied.

### Claim 11 (depends on 1)

The method according to claim 1, wherein before determining the difference loss between the predicted target image and the second sample image, the method further comprises:
adding noise to the second sample image to obtain a noise image; and
determining a second structural difference loss based on the noise image and the second sample image.

### Claim 12 (depends on 11)

The method according to claim 11, wherein determining the second structural difference loss based on the noise image and the second sample image comprises:
processing the noise image based on the structure analysis network to determine at least one third structural feature of at least one position in the noise image;
processing the second sample image based on the structure analysis network to determine the at least one second structural feature of the at least one position in the second sample image; and
determining the second structural difference loss between the noise image and the second sample image based on the at least one third structural feature and the at least one second structural feature.

### Claim 13 (depends on 12)

The method according to claim 12, wherein processing the noise image based on the structure analysis network to determine the at least one third structural feature of the at least one position in the noise image comprises:
processing the noise image based on the structure analysis network to obtain at least one third feature map in at least one scale of the noise image; and
obtaining, for each of the at least one third feature map, the at least one third structural feature of the noise image based on a cosine distance between a feature of each of at least one position in the third feature map and a feature of an adjacent region to the position, wherein each position in the third feature map corresponds to one third structural feature, and the adjacent region feature is each feature in a region centered on the position and comprising at least two positions.

### Claim 14 (depends on 12)

The method according to claim 12, wherein the each position in the third feature map has a correspondence with the each position in the second feature map;
wherein determining the second structural difference loss between the noise image and the second sample image based on the at least one third structural feature and the at least one second structural feature comprises:
calculating a distance between the third structural feature and the second structural feature respectively corresponding to positions having a correspondence; and
determining the second structural difference loss between the noise image and the second sample image based on distances between all third structural features and second structural features corresponding to the noise image.

### Claim 15 (depends on 11)

The method according to claim 11, wherein performing the adversarial training on the image generation network and the structure analysis network based on the difference loss to obtain the trained image generation network comprises:
adjusting, in a third iteration, the network parameter in the image generation network based on the first structural difference loss, the feature loss, and the color loss;
adjusting, in a fourth iteration, the network parameter in the structure analysis network based on the first structural difference loss and the second structural difference loss, wherein the third iteration and the fourth iteration are two continuously-executed iterations; and
obtaining the trained image generation network when the training stopping condition is satisfied.

### Claim 16 (depends on 4)

The method according to claim 4, wherein after processing the predicted target image based on the structure analysis network to determine the at least one first structural feature of the at least one position in the predicted target image, the method further comprises:
performing image reconstruction processing on the at least one first structural feature based on an image reconstruction network to obtain a first reconstructed image; and
determining a first reconstruction loss based on the first reconstructed image and the predicted target image.

### Claim 17 (depends on 16)

The method according to claim 16, wherein after processing the second sample image based on the structure analysis network to determine the at least one second structural feature of the at least one position in the second sample image, the method further comprises:
performing image reconstruction processing on the at least one second structural feature based on an image reconstruction network to obtain a second reconstructed image; and
determining a second reconstruction loss based on the second reconstructed image and the second sample image.

### Claim 18 (depends on 17)

The method according to claim 17, wherein performing the adversarial training on the image generation network and the structure analysis network based on the difference loss to obtain the trained image generation network comprises:
adjusting, in a fifth iteration, the network parameter in the image generation network based on the first structural difference loss, the feature loss, and the color loss;
adjusting, in a sixth iteration, the network parameter in the structure analysis network based on the first structural difference loss, the second structural difference loss, the first reconstruction loss, and the second reconstruction loss, wherein the fifth iteration and the sixth iteration are two continuously-executed iterations; and
obtaining the trained image generation network when the training stopping condition is satisfied.

### Claim 19

An electronic device, comprising:
a processor, and
a memory configured to store processor-executable instructions,
wherein the processor is configured to:
obtain a sample image, wherein the sample image comprises a first sample image and a second sample image corresponding to the first sample image;
process the first sample image based on an image generation network to obtain a predicted target image;
determine a difference loss between the predicted target image and the second sample image; and
train the image generation network based on the difference loss to obtain a trained image generation network.

### Claim 20

A non-transitory computer storage medium, having computer-readable instructions stored therein, wherein the instructions, when being executed, cause to perform operations of the method for training an image generation network, comprising:
obtaining a sample image, wherein the sample image comprises a first sample image and a second sample image corresponding to the first sample image;
processing the first sample image based on an image generation network to obtain a predicted target image;
determining a difference loss between the predicted target image and the second sample image; and
training the image generation network based on the difference loss to obtain a trained image generation network.
