Systems and methods of addressing compression artifacts in video streams
AssigneeNintendo Co., Ltd.
InventorsPierre CORDANI, Christophe AGUETTAZ, Damien RAFFARD, Sebastien GUIDO
Life of this patent
Estimating the term…
Abstract
The technology described herein relates to video streams and image or video compression. More particularly, the technology described herein relates to addressing or removing (e.g., in real-time) image or video compression artifacts that are included in a data stream.
Claims21
A method of training a machine learning model for processing compression artifacts included in video game images, the method comprising:
- retrieving input images that are each associated with a corresponding ground truth image;
- generating modified input images by applying, for each of the input images, a blurring process to the input image that is based on the corresponding ground truth image for that input image;
- generating training data based on compressing, using a compression process, the modified input images, the training data comprising training images that each are associated with a corresponding ground truth image, wherein multiple different ones of the training images are associated with the same corresponding ground truth image and compressed with varying compression quality;
- executing a training process to train a machine learning model based on the generated training data and a loss function, wherein the training processing includes:
- a) encoding feature data based on the training data,
- b) generating first and second prediction heads based at least in part on the encoded feature data, and
- c) calculating a loss for the training process that uses outputs from the first and second prediction heads in the loss function;
- and
- based on convergence of the machine learning model, storing one or more trained machine learning models.
A method of decreasing compression artifacts included in video displayed on a computing system, the method comprising:
- obtaining compressed image data of a compressed video stream;
- obtaining, based on YUV color space of the compressed image data, source luma data and source chroma data;
- performing machine learning inference with the source luma data against a trained machine learning model to generate output luma data;
- generating output chroma data by interpolating the source chroma data using the output luma data;
- generating output image data based on the output luma data and the output chroma data;
- and
- rendering, to a display, an output video stream based on the generated output image data.
A method of reducing or removing banding artifacts included in videos of a video game displayed on a computing system, the method comprising:
- executing a video game application;
- generating, based on a virtual space for the video game, video game image data;
- during execution of the video game application, determining access to at least a first video asset of the video game that is stored at an original resolution;
- based on determination of access to at least the first video asset of the video game, performing machine learning inference with image data of the first video asset against a first trained machine learning model to generate band-corrected image data;
- performing machine learning inference with the band-corrected image data against the second trained machine learning model to generate upscaled image data of the first video asset
- applying blue noise dithering to image frames of the upscaled version of the first video asset to generate dithered image data;
- and
- outputting, based on the dithered image data, an upscaled version of the first video asset that has reduced banding artifacts.
Dependencies are inferred from the claim text.