# US 10,319,076 B2 · Producing higher-quality samples of natural images

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,319,076 B2
- Document ID: US10319076B2
- Status: Granted
- Kind: B2
- Application number: 15/624,429
- Filed: 2017-06-15
- Published: 2017-12-21
- Granted: 2019-06-11
- Earliest claimed: 2016-06-16
- Assignees: Facebook, Inc.
- Applicants: Facebook, Inc.
- Inventors: Emily Denton; Soumith Chintala; Arthur David Szlam; Robert D. Fergus
- Web page: https://patentgrep.com/patent/US10319076B2

## CPC

- G06T 5/60: using machine learning, e.g. neural networks
- G06T 2207/20016: Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

In one embodiment, a method includes accessing a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid. Each GAN may comprise a generative model Gk and a discriminative model Dk. At each level k, the generative model Gk may take as input a noise vector zk and may output a generated image {tilde over (h)}k. At each level k, the discriminative model Dk may take as input either the generated image {tilde over (h)}k or a real image hk, and may output a probability that the input was the real image hk. The method may further include generating a sample image Ĩk from the generated images {tilde over (h)}k, wherein the sample image is based on the probabilities outputted by each of the discriminative models Dk and the generated images {tilde over (h)}k. The method may further include providing the sample image Ĩk for display.

## Claims

### Claim 1

A method comprising, by one or more computing devices:
accessing a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid, wherein each GAN comprises a generative model Gk and a discriminative model Dk, wherein, at each level k:
the generative model G, takes as input a noise vector zk and outputs a generated image {tilde over (h)}k; and
the discriminative model Dk takes as input either the generated image {tilde over (h)}k or a real image hk drawn from a set of training data, and outputs a probability that the input was the real image hk, wherein the set of training data comprises:
a first set of color images of a plurality of objects belonging to a plurality of different classes, wherein the images in the first set are tightly cropped around the respective objects; and
a second set of color images of a plurality of natural scene types, wherein the images in the second set are not tightly cropped around any object:
generating a sample image Ĩk from the generated images {tilde over (h)}k, wherein the sample image is based at least in part on the probabilities outputted by each of the discriminative models Dk at each level k of the Laplacian pyramid and the generated images {tilde over (h)}k; and
providing the sample image Ĩk for display on a display screen of a client device of a user of a communications network.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the sample image Ĩk from the generated images {tilde over (h)}k further comprises:
applying an upsampling operator u(.) to at least one of the generated images {tilde over (h)}k; and
combining the upsampled image {tilde over (h)}k with an image Ik+1 generated by a generative model Gk+1, wherein the image Ik+1 was generated at level k+1 in the Laplacian pyramid.

### Claim 3 (depends on 2)

The method of claim 2, wherein the sample image Ĩk is generated using the equation Ĩk=u(Ĩk+1)+{tilde over (h)}k.

### Claim 4 (depends on 1)

The method of claim 1, wherein the generated images {tilde over (h)}k are generated in a coarse-to-fine fashion.

### Claim 5 (depends on 1)

The method of claim 1, wherein each level k in the Laplacian pyramid corresponds to a generated image {tilde over (h)}k that comprises a particular number of pixels, wherein as k increases, the number of pixels in {tilde over (h)}k decreases.

### Claim 6 (depends on 1)

The method of claim 1, wherein the generative model Gk and the discriminative model Dk each take an additional vector of information as input.

### Claim 7 (depends on 6)

The method of claim 6, wherein the additional vector of information is a generated image lk created by a conditional GAN model.

### Claim 8

One or more computer-readable non-transitory storage media embodying software that is operable when executed to:
access a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid, wherein each GAN comprises a generative model G and a discriminative model Dk, wherein, at each level k:
the generative model Gk takes as input a noise vector zk and outputs a generated image {tilde over (h)}k; and
the discriminative model Dk, takes as input either the generated image {tilde over (h)}k or a real image hk drawn from a set of training data, and outputs a probability that the input was the real image hk, wherein the set of training data comprises:
a first set of color images of a plurality of objects belonging to a plurality of different classes, wherein the images in the first set are tightly cropped around the respective objects; and
a second set of color images of a plurality of natural scene types, wherein the images in the second set are not tightly cropped around any object;
generate a sample image Ĩk from the generated images {tilde over (h)}k, wherein the sample image is based at least in part on the probabilities outputted by each of the discriminative models Dk at each level k of the Laplacian pyramid and the generated images {tilde over (h)}k; and
provide the sample image Ĩk for display on a display screen of a client device of a user of a communications network.

### Claim 9 (depends on 8)

The media of claim 8, wherein generating the sample image Ĩk from the generated images {tilde over (h)}k, further comprises:
applying an upsampling operator u(.) to at least one of the generated images {tilde over (h)}k; and
combining the upsampled image {tilde over (h)}k with an image Ik+1 generated by a generative model Gk+1, wherein the image Ik+1 was generated at level k+1 in the Laplacian pyramid.

### Claim 10 (depends on 9)

The media of claim 9, wherein the sample image Ĩk is generated using the equation Ĩk=u(Ĩk+1)+{tilde over (h)}k.

### Claim 11 (depends on 8)

The media of claim 8, wherein the generated images {tilde over (h)}k are generated in a coarse-to-fine fashion.

### Claim 12 (depends on 8)

The media of claim 8, wherein each level k in the Laplacian pyramid corresponds to a generated image {tilde over (h)}k that comprises a particular number of pixels, wherein as k increases, the number of pixels in {tilde over (h)}k decreases.

### Claim 13 (depends on 8)

The media of claim 8, wherein the generative model Gk and the discriminative model Dk each take an additional vector of information as input.

### Claim 14 (depends on 13)

The media of claim 13, wherein the additional vector of information is a generated image lk created by a conditional GAN model.

### Claim 15

A system comprising:
one or more processors; and
one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by one or more of the processors to cause the system to:
access a plurality of generative adversarial networks (GANs) that are each applied to a particular level k of a Laplacian pyramid, wherein each GAN comprises a generative model Gk and a discriminative model Dk, wherein, at each level k:
the generative model Gk takes as input a noise vector zk and outputs a generated image {tilde over (h)}k; and
the discriminative model Dk takes as input either the generated image {tilde over (h)}k or a real image hk drawn from a set of training data, and outputs a probability that the input was the real image hk, wherein the set of training data comprises:
a first set of color images of a plurality of objects belonging to a plurality of different classes, wherein the images in the first set are tightly cropped around the respective objects; and
a second set of color images of a plurality of natural scene types, wherein the images in the second set are not tightly cropped around any object:
generate a sample image Ĩk from the generated images {tilde over (h)}k, wherein the sample image is based at least in part on the probabilities outputted by each of the discriminative models Dk at each level k of the Laplacian pyramid and the generated images {tilde over (h)}k; and
provide the sample image Ĩk for display on a display screen of a client device of a user of a communications network.

### Claim 16 (depends on 15)

The system of claim 15, wherein generating the sample image Ĩk from the generated images {tilde over (h)}k further comprises:
applying an upsampling operator u(.) to at least one of the generated images {tilde over (h)}k; and
combining the upsampled image {tilde over (h)}k with an image Ik+1 generated by a generative model Gk+1, wherein the image Ik+1 was generated at level k+1 in the Laplacian pyramid.

### Claim 17 (depends on 16)

The system of claim 16, wherein the sample image Ĩk is generated using the equation Ĩk=u(Ĩk+1)+{tilde over (h)}k.

### Claim 18 (depends on 15)

The system of claim 15, wherein the generated images {tilde over (h)}k are generated in a coarse-to-fine fashion.

### Claim 19 (depends on 15)

The system of claim 15, wherein each level k in the Laplacian pyramid corresponds to a generated image {tilde over (h)}k that comprises a particular number of pixels, wherein as k increases, the number of pixels in {tilde over (h)}k decreases.

### Claim 20 (depends on 15)

The system of claim 15, wherein the generative model Gk and the discriminative model Dk each take an additional vector of information as input.