# US 2021/0279841 A1 · Techniques to use a neural network to expand an image

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279841 A1
- Document ID: US20210279841A1
- Status: Published application
- Kind: A1
- Application number: 16/813,589
- Filed: 2020-03-09
- Published: 2021-09-09
- Earliest claimed: 2020-03-09
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Guilin Liu; Andrew Tao; Bryan Christopher Catanzaro; Ting-Chun Wang; Zhiding Yu; Shiqiu Liu; Fitsum Reda; Karan Sapra; Brandon Rowlett
- Web page: https://patentgrep.com/patent/US20210279841A1

## CPC

- G06T 3/4038: Image mosaicing, e.g. composing plane images from plane sub-images
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G06T 3/4046: using neural networks
- G06T 7/40: Analysis of texture (depth or shape recovery from texture G06T7/529)
- G06T 11/10: Texturing; Colouring; Generation of textures or colours (retouching, inpainting or scratch removal G06T5/77)
- G06V 10/454: {Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]}
- G06V 10/54: relating to texture
- G06V 10/776: Validation; Performance evaluation
- G06V 10/82: using neural networks
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 3/063: using electronic means
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING

## Abstract

Apparatuses, systems, and techniques for texture synthesis from small input textures in images using convolutional neural networks. In at least one embodiment, one or more convolutional layers are used in conjunction with one or more transposed convolution operations to generate a large textured output image from a small input textured image while preserving global features and texture, according to various novel techniques described herein.

## Claims

### Claim 1

A processor comprising:
one or more circuits to use one or more neural networks to generate a second image based, at least in part, on one or more feature maps corresponding to a first image, wherein the first image is smaller than the second image.

### Claim 2 (depends on 1)

The processor of claim 1, wherein:
the one or more feature maps are generated by the one or more neural networks from the first image;
one or more shifted feature maps are generated from the one or more feature maps;
one or more weights are computed based, at least in part, on features shared between the one or more shifted feature maps and the one or more feature maps;
one or more combined feature maps are generated based, at least in part, on combining the one or more shifted feature maps and the one or more feature maps according to the one or more weights; and
the second image is generated by aggregating and upsampling the one or more combined feature maps.

### Claim 3 (depends on 2)

The processor of claim 2, wherein each of the one or more feature maps is scaled from the first image using one or more convolutional layers.

### Claim 4 (depends on 2)

The processor of claim 2, wherein the one or more weights are computed based, at least in part, on which corresponding feature of the one or more shifted feature maps and the one or more feature maps is more prominent.

### Claim 5 (depends on 2)

The processor of claim 2, wherein upsampling comprises zero-padding a smaller first feature map into a larger second feature map.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the first image contains a texture and the second image contains the texture of the first image.

### Claim 7

A system comprising:
one or more processors to train one or more neural networks to generate a second image based, at least in part, on one or more feature maps corresponding to a first image, wherein the first image is smaller than the second image.

### Claim 8 (depends on 7)

The system of claim 7, wherein:
the one or more feature maps are scaled from the first image by applying one or more convolutional layers;
one or more weights are computed indicating a similarity between one or more shifted feature maps and the one or more feature maps;
one or more combined feature maps are generated by applying a summation on the one or more shifted feature maps and the one or more feature maps according to the one or more weights; and
the second image is generated based, at least in part, on aggregating the one or more combined feature maps.

### Claim 9 (depends on 8)

The system of claim 8, wherein the one or more shifted feature maps are determined by shifting the one or more feature maps according to a width and a height corresponding to each of the one or more feature maps.

### Claim 10 (depends on 8)

The system of claim 8, wherein the one or more weights indicate whether a first feature of the one or more feature maps or a second feature of the one or more shifted feature maps is to be included during aggregation of the one or more combined feature maps.

### Claim 11 (depends on 8)

The system of claim 8, wherein aggregating the one or more combined feature maps into the second image comprises upsampling each of the one or more combined feature maps.

### Claim 12 (depends on 11)

The system of claim 11, wherein upsampling each of the one or more combined feature maps comprises zero-padding a smaller first feature map into a larger second feature map.

### Claim 13 (depends on 7)

The system of claim 7, wherein the first image contains a texture and the second image comprises the texture of the first image.

### Claim 14 (depends on 7)

The system of claim 7, wherein the one or more neural networks are trained using a generative adversarial network.

### Claim 15 (depends on 14)

The system of claim 14, wherein one or more loss values are determined by the generative adversarial network and backpropagated to one or more convolutional layers in the one or more neural networks.

### Claim 16

A machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
generate a second image based, at least in part, on one or more feature maps corresponding to a first image using one or more neural networks, wherein the first image is smaller than the second image.

### Claim 17 (depends on 16)

The machine-readable medium of claim 16, wherein:
the one or more feature maps are generated the one or more neural networks from the first image;
one or more weights are computed indicating a similarity between one or more shifted feature maps and the one or more feature maps;
one or more combined feature maps are generated based, at least in part, on combining the one or more shifted feature maps and the one or more feature maps according to the one or more weights; and
the second image is generated by aggregating the one or more combined feature maps.

### Claim 18 (depends on 17)

The machine-readable medium of claim 17, wherein each of the one or more feature maps is scaled from the first image using one or more convolutional layers.

### Claim 19 (depends on 17)

The machine-readable medium of claim 17, wherein the one or more shifted feature maps are determined by shifting the one or more feature maps according to a width and a height corresponding to each of the one or more feature maps.

### Claim 20 (depends on 17)

The machine-readable medium of claim 17, wherein the one or more weights are computed based, at least in part, on which corresponding feature of the one or more shifted feature maps and the one or more feature maps is to be retained.

### Claim 21 (depends on 17)

The machine-readable medium of claim 17, wherein aggregating the one or more combined feature maps into the second image includes zero-padding a smaller first feature map into a larger second feature map.

### Claim 22 (depends on 16)

The machine-readable medium of claim 16, wherein the first image contains a texture and the second image contains the texture of the first image.

### Claim 23

A method comprising:
training one or more neural networks to generate a second image based, at least in part, on one or more feature maps corresponding to a first image, wherein the first image is smaller than the second image.

### Claim 24 (depends on 23)

The method of claim 23, wherein:
the one or more feature maps are scaled from the first image by applying one or more convolutional layers;
one or more weights are computed indicating a similarity between one or more shifted feature maps and the one or more feature maps;
one or more combined feature maps are generated by applying a summation on the one or more shifted feature maps and the one or more feature maps according to the one or more weights; and
the second image is generated by aggregating the one or more combined feature maps.

### Claim 25 (depends on 24)

The method of claim 24, wherein the one or more shifted feature maps are determined by shifting the one or more feature maps according to one or more dimensions for each of the one or more feature maps.

### Claim 26 (depends on 24)

The method of claim 24, wherein the one or more weights indicate whether a first feature of the one or more feature maps or a second feature of the one or more shifted feature maps is more prominent.

### Claim 27 (depends on 24)

The method of claim 24, wherein aggregating the one or more combined feature maps into the second image comprises increasing dimensions for each of the one or more combined feature maps.

### Claim 28 (depends on 27)

The method of claim 27, wherein increasing dimensions for each of the one or more combined feature maps comprises zero-padding a smaller first feature map into a larger second feature map.

### Claim 29 (depends on 23)

The method of claim 23, wherein the first image contains a texture and the second image comprises the texture of the first image.

### Claim 30 (depends on 23)

The method of claim 23, wherein the one or more neural networks are trained using a generative adversarial network.

### Claim 31 (depends on 30)

The method of claim 30, wherein one or more loss values are determined based on one or more components of the generative adversarial network and each of the one or more loss values are used to update one or more weights in one or more nodes of the one or more convolutional neural networks.