# US 2022/0284546 A1 · Iterative multiscale image generation using neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0284546 A1
- Document ID: US20220284546A1
- Status: Published application
- Kind: A1
- Application number: 17/751,359
- Filed: 2022-05-23
- Published: 2022-09-08
- Earliest claimed: 2017-02-24
- Assignees: DeepMind Technologies Limited
- Applicants: DeepMind Technologies Limited
- Inventors: Nal Emmerich Kalchbrenner; Daniel Belov; Sergio Gomez Colmenarejo; Aaron Gerard Antonius van den Oord; Ziyu Wang; Joao Ferdinando Gomes de Freitas; Scott Ellison Reed
- Web page: https://patentgrep.com/patent/US20220284546A1

## CPC

- G06T 3/4046: using neural networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G06N 20/00: Machine learning
- G06T 3/4076: using the original low-resolution images to iteratively correct the high-resolution images
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of generating an output image having an output resolution of N pixels×N pixels, each pixel in the output image having a respective color value for each of a plurality of color channels, the method comprising: obtaining a low-resolution version of the output image; and upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations: obtaining a current version of the output image having a current K×K resolution; and processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having a 2K×2K resolution.

## Claims

### Claim 1

(canceled)

### Claim 2

A method of generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the method comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution,
wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes columns of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes rows of pixels from the updated version of the output image.

### Claim 3 (depends on 2)

The method of claim 2, wherein obtaining the low-resolution version comprises:
generating the low-resolution version using an image generation machine learning model.

### Claim 4 (depends on 3)

The method of claim 3, wherein the image generation machine learning model is an autoregressive image generation machine learning model.

### Claim 5 (depends on 3)

The method of claim 3, wherein the output image is conditioned on an input context and wherein the image generation machine learning model is configured to generate the low-resolution version conditioned on the input context.

### Claim 6 (depends on 5)

The method of claim 5, wherein each convolutional neural network in each set of convolutional neural networks is conditioned on the input context.

### Claim 7 (depends on 1)

The method of claim 1, wherein processing the current version of the output image using the set of convolutional neural networks that are specific to the current resolution to generate the updated version of the output image comprises:
processing the current version of the output image using the first convolutional neural network to generate the first output image;
generating the intermediate version by merging the current version and the first output image;
processing the intermediate version using the second convolutional neural network to generate the second output image; and
generating the updated version by merging the intermediate version and the second output image.

### Claim 8 (depends on 7)

The method of claim 7, wherein merging the current version and the first output image comprises:
generating an intermediate image that includes K columns of pixels from the current image and K columns of pixels from the first output image by alternating columns of pixels from the current version with columns of pixels from the first output image.

### Claim 9 (depends on 7)

The method of claim 7, wherein merging the intermediate version and the second output image comprises:
generating an updated image that includes a plurality of rows of pixels from the intermediate version and a plurality of rows of pixels from the second output image by alternating rows of pixels from the intermediate version with rows of pixels from the second output image.

### Claim 10 (depends on 7)

The method of claim 7,
wherein the one or more channels are ordered according to a channel order,
wherein the first convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the first output image conditioned (i) on the current version and (ii) on values for pixels in the first output image for any channels before the channel in the channel order and (iii) not on values for pixels in the first output image for any channels that are after the channel in the channel order, and
wherein the second convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the second output image conditioned (i) on the intermediate version and (ii) on values for pixels in the second output image for any channels before the channel in the channel order and (iii) not on values for pixels in the second output image for any channels that are after the channel in the channel order.

### Claim 11 (depends on 10)

The method of claim 10, wherein processing the current version using the first convolutional neural network to generate the first output image comprises:
iteratively processing the current version and values from the first output image that have already been generated to generate the first output image, and
wherein processing the intermediate version using the second convolutional neural network to generate the second output image comprises:
iteratively processing the intermediate version and the values from the second output image that have already been generated to generate the second output image.

### Claim 12

A method of generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the method comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution, wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes rows of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes columns of pixels from the updated version of the output image.

### Claim 13 (depends on 12)

The method of claim 12, wherein processing the current version of the output image using the set of convolutional neural networks that are specific to the current resolution to generate the updated version comprises:
processing the current version using the first convolutional neural network to generate the first output image;
generating the intermediate version by merging the current version and the first output image;
processing the intermediate version using the second convolutional neural network to generate the second output image; and
generating the updated version by merging the intermediate version and the second output image.

### Claim 14 (depends on 13)

The method of claim 13, wherein merging the current version and the first output image comprises:
generating an intermediate image that includes rows of pixels from the current image and rows of pixels from the first output image by alternating rows of pixels from the current version with rows of pixels from the first output image.

### Claim 15 (depends on 13)

The method of claim 13, wherein merging the intermediate version and the second output image comprises:
generating an updated image that includes columns of pixels from the intermediate version and columns of pixels from the second output image by alternating columns of pixels from the intermediate version with columns of pixels from the second output image.

### Claim 16 (depends on 12)

The method of claim 12,
wherein the one or more channels are ordered according to a channel order,
wherein the first convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the first output image conditioned (i) on the current version and (ii) on values for pixels in the first output image for any channels before the channel in the channel order and (iii) not on values for pixels in the first output image for any channels that are after the channel in the channel order, and
wherein the second convolutional neural network is configured to, for each of the channels:
generate values for the channel for pixels in the second output image conditioned (i) on the intermediate version and (ii) on values for pixels in the second output image for any channels before the channel in the channel order and (iii) not on values for pixels in the second output image for any channels that are after the channel in the channel order.

### Claim 17 (depends on 16)

The method of claim 16, wherein processing the current version using the first convolutional neural network to generate the first output image comprises:
iteratively processing the current version and values from the first output image that have already been generated to generate the first output image, and
wherein processing the intermediate version using the second convolutional neural network to generate the second output image comprises:
iteratively processing the intermediate version and the values from the second output image that have already been generated to generate the second output image.

### Claim 18 (depends on 12)

The method of claim 12, wherein obtaining the low-resolution version comprises:
generating the low-resolution version using an image generation machine learning model.

### Claim 19 (depends on 18)

The method of claim 18, wherein the image generation machine learning model is an autoregressive image generation machine learning model.

### Claim 20 (depends on 19)

The method of claim 19, wherein the output image is conditioned on an input context and wherein the image generation machine learning model is configured to generate the low-resolution version conditioned on the input context.

### Claim 21 (depends on 20)

The method of claim 20, wherein each convolutional neural network in each set of convolutional neural networks is conditioned on the input context.

### Claim 22

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the operations comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution,
wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes columns of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes rows of pixels from the updated version of the output image.

### Claim 23 (depends on 22)

The system of claim 22, wherein the operations for processing the current version of the output image using the set of convolutional neural networks that are specific to the current resolution to generate the updated version of the output image comprise:
processing the current version of the output image using the first convolutional neural network to generate the first output image;
generating the intermediate version by merging the current version and the first output image;
processing the intermediate version using the second convolutional neural network to generate the second output image; and
generating the updated version by merging the intermediate version and the second output image.

### Claim 24 (depends on 23)

The system of claim 23, wherein the operations for merging the current version and the first output image comprise:
generating a intermediate image that includes columns of pixels from the current image and columns of pixels from the first output image by alternating columns of pixels from the current version with columns of pixels from the first output image.

### Claim 25 (depends on 23)

The system of claim 23, wherein the operations for merging the intermediate version and the second output image comprises:
generating an updated image that includes rows of pixels from the intermediate version and rows of pixels from the second output image by alternating the rows of pixels from the intermediate version with the rows of pixels from the second output image.

### Claim 26 (depends on 23)

The system of claim 23,
wherein the one or more channels are ordered according to a channel order,
wherein the first convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the first output image conditioned (i) on the current version and (ii) on values for pixels in the first output image for any channels before the channel in the channel order and (iii) not on values for pixels in the first output image for any channels that are after the channel in the channel order, and
wherein the second convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the second output image conditioned (i) on the intermediate version and (ii) on values for pixels in the second output image for any channels before the channel in the channel order and (iii) not on values for pixels in the second output image for any channels that are after the channel in the channel order.

### Claim 27 (depends on 26)

The system of claim 26, wherein the operations for processing the current version using the first convolutional neural network to generate the first output image comprise:
iteratively processing the current version and values from the first output image that have already been generated to generate the first output image, and
wherein processing the intermediate version using the second convolutional neural network to generate the second output image comprises:
iteratively processing the intermediate version and the values from the second output image that have already been generated to generate the second output image.

### Claim 28

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the operations comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution, wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes rows of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes columns of pixels from the updated version of the output image.

### Claim 29 (depends on 28)

The system of claim 28, wherein the operations for processing the current version of the output image using the set of convolutional neural networks that are specific to the current resolution to generate the updated version comprise:
processing the current version using the first convolutional neural network to generate the first output image;
generating the intermediate version by merging the current version and the first output image;
processing the intermediate version using the second convolutional neural network to generate the second output image; and
generating the updated version by merging the intermediate version and the second output image.

### Claim 30 (depends on 29)

The system of claim 29, wherein the operations for merging the current version and the first output image comprise:
generating an intermediate image that includes rows of pixels from the current image and rows of pixels from the first output image by alternating the rows of pixels from the current version with the rows of pixels from the first output image.

### Claim 31 (depends on 29)

The system of claim 29, wherein the operations for merging the intermediate version and the second output image comprise:
generating an updated image that includes columns of pixels from the intermediate version and columns of pixels from the second output image by alternating the columns of pixels from the intermediate version with the columns of pixels from the second output image.

### Claim 32 (depends on 28)

The system of claim 28,
wherein the one or more channels are ordered according to a channel order,
wherein the first convolutional neural network is configured to, for each of the one or more channels:
generate values for the channel for pixels in the first output image conditioned (i) on the current version and (ii) on values for pixels in the first output image for any channels before the channel in the channel order and (iii) not on values for pixels in the first output image for any channels that are after the channel in the channel order, and
wherein the second convolutional neural network is configured to, for each of the channels:
generate values for the channel for pixels in the second output image conditioned (i) on the intermediate version and (ii) on values for pixels in the second output image for any channels before the channel in the channel order and (iii) not on values for pixels in the second output image for any channels that are after the channel in the channel order.

### Claim 33 (depends on 32)

The system of claim 32, wherein processing the current version using the first convolutional neural network to generate the first output image comprises:
iteratively processing the current version and values from the first output image that have already been generated to generate the first output image, and
wherein processing the intermediate version using the second convolutional neural network to generate the second output image comprises:
iteratively processing the intermediate version and the values from the second output image that have already been generated to generate the second output image.

### Claim 34

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the operations comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution,
wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes columns of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes rows of pixels from the updated version of the output image.

### Claim 35

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating an output image having an output resolution, each pixel in the output image having a respective value for each of one or more channels, the operations comprising:
obtaining a low-resolution version of the output image; and
upscaling the low-resolution version of the output image to generate the output image having the output resolution by repeatedly performing the following operations until an image with an output resolution is obtained:
obtaining a current version of the output image having a current resolution; and
processing the current version of the output image using a set of convolutional neural networks that are specific to the current resolution to generate an updated version of the output image having an updated resolution that is higher than the current resolution, wherein the set of convolutional neural networks that are specific to the current resolution comprises:
a first convolutional neural network that is configured to receive a first input comprising the current version of the image and to generate a first output image that includes rows of pixels from an intermediate version of the output image having an intermediate resolution that is higher than the current resolution but lower than the updated resolution, and
a second convolutional neural network that is configured to receive a second input comprising the intermediate version of the output image and to generate a second output image that includes columns of pixels from the updated version of the output image.
