# US 12,217,386 B2 · Image generation using one or more neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,386 B2
- Document ID: US12217386B2
- Status: Granted
- Kind: B2
- Application number: 17/060,836
- Filed: 2020-10-01
- Published: 2022-04-07
- Granted: 2025-02-04
- Earliest claimed: 2020-10-01
- Assignees: NVIDIA Corporation
- Applicants: Nvidia Corporation
- Inventors: Brennan Shacklett; Marco Salvi; Aaron Lefohn
- Web page: https://patentgrep.com/patent/US12217386B2

## CPC

- G06T 3/4046: using neural networks
- A63F 13/69: by enabling or updating specific game elements, e.g. unlocking hidden features, items, levels or versions
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06T 3/4053: based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
- G06T 7/20: Analysis of motion (motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51)
- G06N 3/006: based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
- G06N 3/048: Activation functions
- G06N 3/049: Temporal neural networks, e.g. delay elements, oscillating neurons or pulsed inputs
- G06N 20/00: Machine learning
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- A63F: CARD, BOARD, OR ROULETTE GAMES; INDOOR GAMES USING SMALL MOVING PLAYING BODIES; VIDEO GAMES; GAMES NOT OTHERWISE PROVIDED FOR
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Apparatuses, systems, and techniques are presented to generate images. In at least one embodiment, at least a first optical flow network (OFN) and at least a first reconstruction network (RN) can be used to generate one or more images based, at least in part, upon the OFN and the RN using a shared loss function.

## Claims

### Claim 1

A processor, comprising:
one or more circuits to train one or more optical flow networks (OFNs) and one or more reconstruction networks (RNs) partially dependently after the one or more OFNs and RNs have been trained independently to generate outputs within a first range of values.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the one or more circuits are further to train a first OFN and a first RN together using a shared loss function.

### Claim 3 (depends on 2)

The processor of claim 2, wherein the shared loss function comprises one or more optical flow loss terms and one or more image reconstruction loss terms.

### Claim 4 (depends on 1)

The processor of claim 1, wherein a first OFN and a first RN are part of a single fused network.

### Claim 5 (depends on 4)

The processor of claim 4, wherein the one or more circuits are further to train the single fused network using an initial phase where optical flow loss terms are ignored, and a subsequent phase where a contribution of optical flow terms is gradually increased.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the one or more circuits are to generate one or more images based at least in part upon one or more upscaled images from an image renderer, wherein motion vectors are not provided by the image renderer.

### Claim 7

A system comprising:
one or more processors to train one or more optical flow networks (OFNs) and one or more reconstruction networks (RNs) dependently after the one or more OFNs and RNs have been trained independently to generate outputs within a first range of values.

### Claim 8 (depends on 7)

The system of claim 7, wherein the one or more processors are further to train a first OFN and a first RN together using a shared loss function.

### Claim 9 (depends on 8)

The system of claim 8, wherein the shared loss function comprises one or more optical flow loss terms and one or more image reconstruction loss terms.

### Claim 10 (depends on 7)

The system of claim 7, wherein a first OFN and a first RN are part of a single fused network.

### Claim 11 (depends on 10)

The system of claim 10, wherein the one or more processors are further to train the single fused network using an initial phase where optical flow loss terms are ignored, and a subsequent phase where a contribution of optical flow terms is gradually increased.

### Claim 12 (depends on 7)

The system of claim 7, wherein the one or more processors are to generate one or more images based at least in part upon one or more upscaled images from an image renderer, wherein motion vectors are not provided by the image renderer.

### Claim 13

A method comprising:
training one or more optical flow networks (OFNs) and one or more reconstruction networks (RNs) dependently after the one or more OFNs and RNs have been trained independently to generate outputs within a first range of values.

### Claim 14 (depends on 13)

The method of claim 13, further comprising:
training a first OFN and a first RN together using a shared loss function.

### Claim 15 (depends on 14)

The method of claim 14, wherein the shared loss function comprises one or more optical flow loss terms and one or more image reconstruction loss terms.

### Claim 16 (depends on 13)

The method of claim 13, wherein a first OFN and a first RN are part of a single fused network.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
training the single fused network using an initial phase where optical flow loss terms are ignored, and a subsequent phase where a contribution of optical flow terms is gradually increased.

### Claim 18 (depends on 13)

The method of claim 13, wherein one or more images are generated based at least in part upon one or more upscaled images from an image renderer, wherein motion vectors are not provided by the image renderer.

### Claim 19

A non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors to at least:
train one or more optical flow networks (OFNs) and one or more reconstruction networks (RNs) dependently after the one or more OFNs and RNs have been trained independently to generate outputs within a first range of values.

### Claim 20 (depends on 19)

The non-transitory machine-readable medium of claim 19, wherein the one or more processors are further to train a first OFN and a first RN together using a shared loss function.

### Claim 21 (depends on 20)

The non-transitory machine-readable medium of claim 20, wherein the shared loss function comprises one or more optical flow loss terms and one or more image reconstruction loss terms.

### Claim 22 (depends on 19)

The non-transitory machine-readable medium of claim 19, wherein a first OFN and a first RN are part of a single fused network.

### Claim 23 (depends on 22)

The non-transitory machine-readable medium of claim 22, wherein the one or more processors are further to train the single fused network using an initial phase where optical flow loss terms are ignored, and a subsequent phase where a contribution of optical flow terms is gradually increased.

### Claim 24 (depends on 19)

The non-transitory machine-readable medium of claim 19, wherein one or more images are generated by the one or more OFNs and the one or more RNs based at least in part upon one or more upscaled images from an image renderer, wherein motion vectors are not provided by the image renderer.

### Claim 25

An image generation system, comprising:
one or more processors to train one or more optical flow networks (OFNs) and one or more reconstruction networks (RNs) dependently after the one or more OFNs and RNs have been trained independently to generate outputs within a first range of values; and
memory for storing network parameters for the one or more OFNs and the one or more RNs.

### Claim 26 (depends on 25)

The image generation system of claim 25, wherein the one or more processors are further to train a first OFN and a first RN together using a shared loss function.

### Claim 27 (depends on 26)

The image generation system of claim 26, wherein the shared loss function comprises one or more optical flow loss terms and one or more image reconstruction loss terms.

### Claim 28 (depends on 25)

The image generation system of claim 25, wherein a first OFN and a first RN are part of a single fused network.

### Claim 29 (depends on 28)

The image generation system of claim 28, wherein the one or more processors are further to train the single fused network using an initial phase where optical flow loss terms are ignored, and a subsequent phase where a contribution of optical flow terms is gradually increased.

### Claim 30 (depends on 25)

The image generation system of claim 25, wherein one or more images are generated by the one or more OFNs and the one or more RNs based at least in part upon one or more upscaled images from an image renderer, wherein motion vectors are not provided by the image renderer.