US 2025/0050212 A1 · EFFICIENT SUPER-SAMPLING IN VIDEOS USING HISTORICAL INTERMEDIATE FEATURES

Abstract

Systems and methods for providing a high-resolution gaming experience on typical computer systems, including computer systems without high-end d-GPUs. In particular, systems and methods are provided for optimizing deep learning-based super-sampling methods. A hardware-aware optimization technique for super-sampling machine learning networks uses a subset of intermediate outputs of the machine learning model for the previous game frame for convolution operations on the current frame, thereby reducing compute usage and latency without sacrificing quality of the output. The inputs are concatenated and passed through a convolutional neural network (CNN), such as a U-net-based CNN. The output of the CNN is a high-resolution image frame that can be post-processed to generate a final output. The hardware optimization technique can be implemented in a neural network framework that divides the machine learning inference across available compute resources on the computer platform.

Claims20

Dependent claims

A method comprising:

receiving, at an input channel, input video including a current image frame and a previous image frame;
performing, at a first convolution layer, a first set of convolution operations on the current image frame to generate a first subset of intermediate convolution outputs;
generating a first set of intermediate convolution outputs, including the first subset of intermediate convolution outputs and a previous subset of intermediate convolution outputs from the previous image frame;
performing, at a second convolution layer, a second set of convolution operations on the first set of intermediate convolution outputs;
and
outputting a high-resolution image frame.

3 more elements

An apparatus, comprising:

a computer processor for executing computer program instructions;
and
a non-transitory computer-readable memory storing computer program instructions executable by the computer processor to perform operations comprising:
receiving, at an input channel, input video including a current image frame and a previous image frame;
performing, at a first convolution layer, a first set of convolution operations on the current image frame to generate a first subset of intermediate convolution outputs;
generating a first set of intermediate convolution outputs, including the first subset of intermediate convolution outputs and a previous subset of intermediate convolution outputs from the previous image frame;
performing, at a second convolution layer, a second set of convolution operations on the first set of intermediate convolution outputs;
and
outputting a high-resolution image frame.

6 more elements

Dependencies are inferred from the claim text.

Efficient super-sampling in videos using historical intermediate features

Life of this patent

Efficient super-sampling in videos using historical intermediate features

Life of this patent

Abstract

Claims20