# US 2023/0410375 A1 · Temporally stable data reconstruction with an external recurrent neural network

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0410375 A1
- Document ID: US20230410375A1
- Status: Published application
- Kind: A1
- Application number: 18/225,674
- Filed: 2023-07-24
- Published: 2023-12-21
- Earliest claimed: 2017-07-27
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Marco Salvi; Anjul Patney; Aaron Eliot Lefohn; Donald Lee Brittain
- Web page: https://patentgrep.com/patent/US20230410375A1

## CPC

- G06T 9/002: {using neural networks}
- G06N 3/02: Neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06T 3/16: Spatio-temporal transformations, e.g. video cubism
- G06T 5/60: using machine learning, e.g. neural networks
- G06T 5/70: Denoising; Smoothing
- G06T 2207/20081: Training; Learning
- G06T 2207/20084: Artificial neural networks [ANN]
- G06T 2207/20182: Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method, computer readable medium, and system are disclosed for temporally stable data reconstruction. A sequence of input data including artifacts is received. A first input data frame is processed using layers of a neural network model to produce external state including a reconstructed first data frame that approximates the first input data frame without artifacts. Hidden state generated during processing of the first input data is not provided as an input to the layer to process second input data. The external state is warped, using difference data corresponding to changes between input data frames, to produce warped external state more closely aligned with the second input data frame. The second input data frame is processed, based on the warped external state, using the layers of the neural network model to produce a reconstructed second data frame that approximates the second data frame without artifacts.

## Claims

### Claim at position 1

1-20. (canceled)

### Claim 21

A processor, comprising: one or more circuits to modify one or more neural networks based, at least in part, on motion of one or more pixels between a first frame and a second frame.

### Claim 22 (depends on 21)

The processor of claim 21, wherein the neural network comprises an encoder/decoder neural network.

### Claim 23 (depends on 21)

The processor of claim 21, wherein the one or more neural networks are to combine at least one filter kernel and at least two successive frames.

### Claim 24 (depends on 21)

The processor of claim 21, wherein the one or more neural networks are to apply at least a first portion of at least one filter kernel to reconstructed data and to apply at least a second portion of the at least one filter kernel to the input data.

### Claim 25 (depends on 21)

The processor of claim 21, wherein the one or more neural networks comprise two or more filter kernels to be applied to different respective areas of at least one of the first frame or second frame.

### Claim 26 (depends on 24)

The processor of claim 24, wherein the one or more circuits are to further generate different filter kernels to be used at different respective locations of at least the first frame.

### Claim 27 (depends on 21)

The processor of claim 21, wherein the first and second frames are successive video frames.

### Claim 28

A system, comprising memory to store instructions that, as a result of performance by one or more processors, cause the system to modify one or more neural networks based, at least in part, on motion of one or more pixels between a first frame and a second frame.

### Claim 29 (depends on 28)

The system of claim 28, wherein the neural network comprises an encoder/decoder neural network.

### Claim 30 (depends on 28)

The system of claim 28, wherein the one or more neural networks are to combine at least one filter kernel and at least two successive frames.

### Claim 31 (depends on 28)

The system of claim 28, wherein the one or more neural networks are to apply at least a first portion of at least one filter kernel to reconstructed data and to apply at least a second portion of the at least one filter kernel to the input data.

### Claim 32 (depends on 28)

The system of claim 28, wherein the one or more neural networks comprise two or more filter kernels to be applied to different respective areas of at least one of the first frame or second frame.

### Claim 33 (depends on 31)

The system of claim 31, wherein the one or more circuits are to further generate different filter kernels to be used at different respective locations of at least the first frame.

### Claim 34 (depends on 28)

The system of claim 28, wherein the first and second frames are successive video frames.

### Claim 35

A method comprising modifying one or more neural networks based, at least in part, on motion of one or more pixels between a first frame and a second frame.

### Claim 36 (depends on 35)

The method of claim 35, wherein the one or more neural networks are to combine at least one filter kernel and at least two successive frames.

### Claim 37 (depends on 35)

The method of claim 35, wherein the one or more neural networks are to combine at least one filter kernel and at least two successive frames.

### Claim 38 (depends on 35)

The method of claim 35, wherein the one or more neural networks are to apply at least a first portion of at least one filter kernel to reconstructed data and to apply at least a second portion of the at least one filter kernel to the input data.

### Claim 39 (depends on 35)

The method of claim 35, wherein the one or more neural networks comprise two or more filter kernels to be applied to different respective areas of at least one of the first frame or second frame.

### Claim 40 (depends on 38)

The method of claim 38, one or more circuits are to further generate different filter kernels to be used at different respective locations of at least the first frame.
