# US 2021/0012486 A1 · Image synthesis with generative adversarial network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0012486 A1
- Document ID: US20210012486A1
- Status: Published application
- Kind: A1
- Application number: 16/905,923
- Filed: 2020-06-19
- Published: 2021-01-14
- Earliest claimed: 2019-07-09
- Assignees: Shenzhen Malong Technologies Co., Ltd.
- Applicants: Shenzhen Malong Technologies Co., Ltd.
- Inventors: Yawen HUANG; Weilin HUANG; Matthew Robert SCOTT
- Web page: https://patentgrep.com/patent/US20210012486A1

## CPC

- G06N 3/045: Combinations of networks
- G06T 7/0012: {Biomedical image inspection}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/0475: Generative networks
- G06N 3/048: Activation functions
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 11/00: Two-dimensional [2D] image generation
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06T 2207/10088: Magnetic resonance imaging [MRI]
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/30016: Brain
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

Aspects of the technology described herein provide a system for improved synthesis of a target domain image from a source domain image. A generator that performs the synthesis is formed based on the texture propagation from the first domain to the second domain with a bidirectional generative adversarial network, which is trained for the texture propagation with a shape prior constraint.

## Claims

### Claim 1

An apparatus for synthesizing images, the apparatus comprising a memory having computer programs stored thereon and a processor configured to perform, when executing the computer programs, operations comprising:
generating a target image in a first imaging modality from a source image in a second imaging modality based on a texture propagation operation in a bidirectional generative adversarial network that comprises a first image generator network and a second image generator network, wherein the second image generator network is an inverse of the first image generator network.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the texture propagation operation comprises propagating texture details from the source image to the target image by using a plurality of feature maps of a deep network to preserve local textural details at a plurality of convolutional layers of the deep network.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the plurality of feature maps comprise a first feature map modeling features of a target domain sample, a second feature map modeling features of a synthesized source domain sample, a third feature map modeling features of a source domain sample, and a fourth feature map modeling features of a synthesized target domain.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the operations further comprising:
iteratively modifying the first image generator network and the second image generator network in accordance with an entropy loss that comprises a texture entropy loss term that employs a 1-norm.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the entropy loss further comprises a segmentation cross entropy loss term.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the segmentation cross entropy loss term calculates a cross entropy loss across a set of brain tissue classes including at least one of Cerebrospinal Fluid, Gray Matter, or White Matter.

### Claim 7 (depends on 4)

The apparatus of claim 4, wherein the entropy loss further comprises at least one of a domain matching loss term, a cycle consistency loss term, or a bidirectional loss term.

### Claim 8 (depends on 1)

The apparatus of claim 1, wherein the bidirectional generative adversarial network is trained based on a shape prior constraint.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the shape prior constraint comprises target shape information from a target domain, and source shape information from a source domain.

### Claim 10 (depends on 9)

The apparatus of claim 9, wherein the source domain is a first mode of data collection and the target domain is a second mode of data collection pertaining to subjects with one or more similar attributes.

### Claim 11

A method of training a network for synthesizing images, comprising:
receiving a corpus of source samples in a source domain and a corpus of target samples in a target domain; and
forming a generator network estimate based on texture propagation through bidirectional generative adversarial network estimation using information contained in the corpus of source samples and in the corpus of target samples.

### Claim 12 (depends on 11)

The method of claim 11, further comprising:
propagating texture details from a source image to a target image based on a descriptor to preserve local textural details at convolutional layers.

### Claim 13 (depends on 12)

The method of claim 12, wherein the descriptor comprises a first feature map at a first layer modeling features of a target domain sample, a second feature map at a second layer modeling features of a synthesized source domain sample, a third feature map at a third layer modeling features of a source domain sample, and a fourth feature map at a fourth layer modeling features of a synthesized target domain.

### Claim 14 (depends on 11)

The method of claim 11, wherein the network comprises a first image generator network and a second image generator network, and the method further comprising:
iteratively modifying the first image generator network and the second image generator network in accordance with an entropy loss that comprises a texture entropy loss term.

### Claim 15 (depends on 11)

The method of claim 11, further comprising:
extracting shape information from a target domain sample using a target domain segmentor and from a source domain sample using a source domain segmentor; and
forming the generator network estimate further based on the shape information.

### Claim 16 (depends on 15)

The method of claim 15, wherein the source domain sample comprises a voxel.

### Claim 17 (depends on 11)

The method of claim 11, wherein the source domain is a first mode of data collection and the target domain is a second and distinct mode of data collection pertaining to subjects with one or more similar attributes.

### Claim 18

A system for synthesizing images, comprising:
a generator to operate on a source image to produce a target image, and to propagate texture details from the source image to the target image based on a deep network; and
the deep network to preserve the texture details at convolutional layers of the deep network based on a texture propagation mechanism and a shape prior constraint.

### Claim 19 (depends on 18)

The system of claim 18, wherein the generator comprises a first image generator network and a second image generator network, and the first image generator network and the second image generator network are iteratively modified by processing an input voxel in accordance with an entropy loss that comprises a texture entropy loss term and a segmentation cross entropy loss term.

### Claim 20 (depends on 18)

The system of claim 18, wherein the shape prior constraint comprises source shape information and target shape information, the system further comprising:
a source domain segmentor to extract the source shape information from a source domain sample; and
a target domain segmentor to extract the target shape information from a target domain sample.
