# US 11,308,361 B1 · Checkerboard artifact free sub-pixel convolution

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,361 B1
- Document ID: US11308361B1
- Status: Granted
- Kind: B1
- Application number: 16/028,374
- Filed: 2018-07-05
- Granted: 2022-04-19
- Earliest claimed: 2017-07-07
- Assignees: Twitter, Inc.
- Applicants: Twitter, Inc.
- Inventors: Andrew Aitken; Christian Ledig; Lucas Theis; Jose Caballero; Zehan Wang; Wenzhe Shi
- Web page: https://patentgrep.com/patent/US11308361B1

## CPC

- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06V 10/82: using neural networks
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/73: Deblurring; Sharpening
- G06N 3/047: Probabilistic or stochastic networks
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

An example system includes a processor and a memory. The system performs sub-pixel convolution that is free of checkerboard artifacts. In one example implementation, the system may execute a method that includes initializing one or more parameters of a sub-kernel of a kernel and copying the one or more parameters of the sub-kernel to other sub-kernels of the kernel. The method may further include performing convolution of an input image with the sub-kernels of the kernel and generating a plurality of first output images. A second output image is then generated based on the plurality of first output images.

## Claims

### Claim 1

A method, comprising:
initializing a model including a plurality of channels, each channel including a two-dimensional (2D) convolution kernel, each 2D convolution kernel including two or more sub-kernels where the number of sub-kernels is based on an upscaling factor, and each of the two or more sub-kernels is used to generate one subset of pixels in a super-resolved image, where initializing a channel of the model includes:
initializing parameters of a first sub-kernel the two or more sub-kernels associated with the channel, and
copying the parameters of the first sub-kernel to other sub-kernels of the two or more sub-kernels, wherein as a result of the copying each of the two or more sub-kernels has the initialized parameters of the first sub-kernel;
performing convolution of an input image with the two or more sub-kernels of the kernel and generating a plurality of first output images; and
generating a second output image, as the super-resolved image, based on the plurality of first output images.

### Claim 2 (depends on 1)

The method of claim 1, wherein
the convolution is a sub-pixel convolution, and
consecutive sub-kernels are associated with a sub-pixel space.

### Claim 3 (depends on 2)

The method of claim 2, wherein the initializing of a parameter of the parameters includes assigning a weight to the parameter.

### Claim 4 (depends on 3)

The method of claim 3, wherein the weight is assigned such that the sub-pixel convolution, after the initialization, is identical to a nearest neighbor (NN) resize convolution.

### Claim 5 (depends on 3)

The method of claim 3, wherein the kernel is for a convolutional neural network (CNN) and the method further comprises:
training the CNN and generating the super-resolved image based on the input image.

### Claim 6 (depends on 5)

The method of claim 5, wherein the training further comprises:
adjusting the parameters of a sub-kernel of the kernel; and
repeating the copying, the performing, and the generating.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the second output image based on the plurality of first output images includes re-arranging pixels of the plurality of first output images to generate the second output image.

### Claim 8

A computing device, comprising:
a processor; and
a memory, the memory including instructions configured to cause the processor to:
initialize a model including a plurality of channels, each channel including a two-dimensional (2D) convolution kernel each 2D convolution kernel including two or more sub-kernels where the number of sub-kernels is based on an upscaling factor, and each of the two or more sub-kernels is used to generate one subset of pixels in a super-resolved image, where initializing a channel of the model includes:
initializing parameters of a first sub-kernel of the two or more sub-kernels associated with the channel, and;
copying the parameters of the first sub-kernel to other sub-kernels of the two or more sub-kernels wherein as a result of the copying each of the two or more sub-kernels has the initialized parameters of the first sub-kernel:
perform convolution of an input image with the sub-kernels of the kernel and generate a plurality of first output images; and
generate a second output image, as the super-resolved image, based on the plurality of first output images.

### Claim 9 (depends on 8)

The computing device of claim 8, wherein
the convolution is a sub-pixel convolution, and
consecutive sub-kernels are associated with a sub-pixel space.

### Claim 10 (depends on 9)

The computing device of claim 9, wherein the initializing of a parameter of the parameters includes assigning a weight to the parameter.

### Claim 11 (depends on 10)

The computing device of claim 10, wherein the weight is assigned such that the convolution, after the initialization, is identical to nearest neighbor (NN) resize convolution.

### Claim 12 (depends on 10)

The computing device of claim 10, wherein the kernel is for a convolutional neural network (CNN) and wherein the processor is further configured to:
train the CNN and generating the super-resolved image based on the input image.

### Claim 13 (depends on 12)

The computing device of claim 12, wherein the processor is further configured to:
adjust the parameters of the first sub-kernel; and
repeat the copying, the performing, and the generating.

### Claim 14 (depends on 8)

The computing device of claim 8, wherein generating the second output image based on the plurality of first output images includes re-arranging pixels of the plurality of first output images to generate the second output image.

### Claim 15

A non-transitory computer-readable storage medium having stored thereon computer executable program code which, when executed on a computer system, causes the computer system to perform a method of progressive mesh compression, comprising:
initializing a model including a plurality of channels, each channel including a two-dimensional (2D) convolution kernel, each 2D convolution kernel including two or more sub-kernels where the number of sub-kernels is based on an upscaling factor, and each of the two or more sub-kernels is used to generate one subset of pixels in a super-resolved image, where initializing a channel of the model includes:
initializing parameters of a first sub-kernel the two or more sub-kernels associated with the channel, and;
copying the parameters of the first sub-kernel to other sub-kernels of the two or more sub-kernels, wherein as a result of the copying each of the two-or more sub-kernels has the initialized parameters of the first sub-kernel;
performing convolution of an input image with the sub-kernels of the kernel and generating a plurality of first output images; and
generating a second output image, as the super-resolved image, based on the plurality of first output images.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the initializing of a parameter of the parameters includes assigning a weight to the first sub-kernel for initializing the model as a convolutional neural network (CNN).

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein the weight is assigned such that the convolution, after the initialization, is identical to nearest neighbor (NN) resize convolution.

### Claim 18 (depends on 16)

The computer-readable storage medium of claim 16, wherein the computer-readable storage medium further comprises code for:
training the CNN and generating the super-resolved image based on the input image.

### Claim 19 (depends on 18)

The computer-readable storage medium of claim 18, wherein the training includes:
adjusting the parameters of the first sub-kernel of the kernel; and
repeating the copying, the performing, and the generating.

### Claim 20 (depends on 15)

The computer-readable storage medium of claim 15, wherein generating the second output image based on the plurality of first output images includes re-arranging pixels of the plurality of first output images to generate the second output image.