# US 12,001,956 B2 · Optimizing supervised generative adversarial networks via latent space regularizations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,001,956 B2
- Document ID: US12001956B2
- Status: Granted
- Kind: B2
- Application number: 18/319,109
- Filed: 2023-05-17
- Published: 2023-09-14
- Granted: 2024-06-04
- Earliest claimed: 2019-04-30
- Assignees: Agora Lab, Inc.
- Applicants: Agora Lab, Inc.
- Inventors: Sheng Zhong
- Web page: https://patentgrep.com/patent/US12001956B2

## CPC

- G06N 3/045: Combinations of networks
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06N 20/00: Machine learning
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

A method for image generation based on a Generative Adversarial Network (GAN) including a generator, a discriminator, and an encoder, wherein outputs of the generator are mapped, by the encoder, to a latent space adaptable to manipulate at least one characteristics of images generated by the GAN, the method including generating, by the encoder, a first encoding E(Y) of a target image Y and a second encoding E(G(Z)) of a generated image G(Z) corresponding to the target image Y, wherein the first and second encodings E(Y) and E(G(Z)) map Y and G(Z) to the latent space having a lower dimension than dimensionality of Y and G(Z), wherein the encoder is trained to minimize the differences between the first and second encodings E(Y) and E(G(Z)), and the generator is trained by using the first and second encodings E(Y) and E(G(Z)) as part of a loss function.

## Claims

### Claim 1

An apparatus comprising:
at least one processor; and
at least one memory, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to implement a Generative Adversarial Network (GAN), the GAN comprising a generator, a discriminator, and an encoder, wherein outputs of the generator are mapped, by the encoder, to a latent space adaptable to manipulate at least one characteristics of images generated by the GAN, and to train the GAN by:
generating, by the encoder, a first encoding E(Y) of a target image Y and a second encoding E(G(Z)) of a generated image G(Z) corresponding to the target image Y, wherein the generated image G(Z) corresponding to the target image Y is generated by the generator, the first encoding E(Y) and the second encoding E(G(Z)) map the target image Y and the generated image G(Z) to the latent space having a lower dimension than dimensionality of the target image Y and the generated image G(Z), and a first loss function between the first encoding E(Y) and the second encoding E(G(Z)) is minimized for the encoder; and
training the generator using the first encoding E(Y) and the second encoding E(G(Z)) as part of a second loss function for the generator.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein a same set of weights are used for first encoding E(Y) of the target image Y and the second encoding E(G(Z)) of the generated image G(Z) corresponding to the target image Y, and the same set of weights are updated after one or more iterations of training the GAN.

### Claim 3 (depends on 1)

The apparatus of claim 1, wherein the encoder comprises a machine learning model.

### Claim 4 (depends on 1)

The apparatus of claim 1, wherein the encoder comprises a convolutional neural network.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the discriminator, the generator, and the encoder are trained, in sequential order, in each iteration of training the GAN.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein for each iteration of training the GAN, the generator and the encoder are updated simultaneously after the discriminator has been updated.

### Claim 7 (depends on 1)

The apparatus of claim 1, wherein the encoder is adapted from at least one of a first few layers of a VGG model, or one or more upscaling layers following the first few layers of the VGG model.

### Claim 8

A method for image generation based on a Generative Adversarial Network (GAN), the GAN comprising a generator, a discriminator, and an encoder, wherein outputs of the generator are mapped, by the encoder, to a latent space adaptable to manipulate at least one characteristics of images generated by the GAN, the method comprising:
generating, by the encoder, a first encoding E(Y) of a target image Y and a second encoding E(G(Z)) of a generated image G(Z) corresponding to the target image Y, wherein the generated image G(Z) corresponding to the target image Y is generated by the generator, the first encoding E(Y) and the second encoding E(G(Z)) map the target image Y and the generated image G(Z) to the latent space having a lower dimension than dimensionality of the target image Y and the generated image G(Z), and a first loss function between the first encoding E(Y) and the second encoding E(G(Z)) is minimized for the encoder; and
training the generator using the first encoding E(Y) and the second encoding E(G(Z)) as part of a second loss function for the generator.

### Claim 9 (depends on 8)

The method of claim 8, wherein a same set of weights are used for first encoding E(Y) of the target image Y and the second encoding E(G(Z)) of the generated image G(Z) corresponding to the target image Y, and the same set of weights are updated after one or more iterations of training the GAN.

### Claim 10 (depends on 8)

The method of claim 8, wherein the encoder comprises a machine learning model.

### Claim 11 (depends on 8)

The method of claim 8, wherein the encoder comprises a convolutional neural network.

### Claim 12 (depends on 8)

The method of claim 8, wherein the discriminator, the generator, and the encoder are trained, in sequential order, in each iteration of training the GAN.

### Claim 13 (depends on 8)

The method of claim 8, wherein for each iteration of training the GAN, the generator and the encoder are updated simultaneously after the discriminator has been updated.

### Claim 14 (depends on 8)

The method of claim 8, wherein the encoder is adapted from at least one of a first few layers of a VGG model, or one or more upscaling layers following the first few layers of the VGG model.

### Claim 15

A method for image generation based on a Generative Adversarial Network (GAN), the GAN comprising a generator, a discriminator, and an encoder, wherein outputs of the generator are mapped, by the encoder, to a latent space adaptable to manipulate at least one characteristics of images generated by the GAN, the method comprising:
generating, by the encoder, a first encoding E(Y) of a target image Y and a second encoding E(G(Z)) of a generated image G(Z) corresponding to the target image Y, wherein the generated image G(Z) corresponding to the target image Y is generated by the generator, and the first and second encodings E(Y) and E(G(Z)) map the target image Y and the generated image G(Z) to the latent space having a lower dimension than dimensionality of the target image Y and the generated image G(Z), wherein the encoder is trained to minimize the differences between the first and second encodings E(Y) and E(G(Z)), and the generator is trained by using the first and second encodings E(Y) and E(G(Z)) as part of a loss function of the generator.

### Claim 16 (depends on 15)

The method of claim 15, wherein the discriminator is trained to distinguish which of the target image Y and the generated image G(Z) corresponding to the target image Y is real data.

### Claim 17 (depends on 15)

The method of claim 15, wherein a same set of weights are used for first encoding E(Y) of the target image Y and the second encoding E(G(Z)) of the generated image G(Z) corresponding to the target image Y, and the same set of weights are updated after one or more iterations of training the GAN.

### Claim 18 (depends on 15)

The method of claim 15, wherein the encoder comprises a machine learning model.

### Claim 19 (depends on 15)

The method of claim 15, wherein the encoder comprises a convolutional neural network.

### Claim 20 (depends on 15)

The method of claim 15, wherein at least one of:
the discriminator, the generator, and the encoder are trained, in sequential order, in each iteration of training the GAN; or
for each iteration of training the GAN, the generator and the encoder are updated simultaneously after the discriminator has been updated.