Systems and methods for machine learned image conversion
AssigneeNintendo Co., Ltd.
InventorsAlexandre Delattre, Théo Charvet, Raphaël Poncet
Life of this patent
Estimating the term…
Abstract
A computer system is provided for converting images through use of a trained neural network. A source image is divided into blocks and context data is added to each pixel block. The context blocks are split into channels and each channel from the same context block is added to the same activation matrix. The action matrix is then executed against a trained neural network to produce a changed activation matrix. The changed activation matrix is then used to generate a converted image.
Claims20
A computer system for upconverting images in real-time for display on a display device, the computer system comprising:
- a processing system that includes a central processing unit (CPU) and a graphical processing unit (GPU), the GPU including a plurality of processing cores, the processing system comprising computer-executable instructions that are configured to cause at least one hardware processor to perform operations comprising:
- executing a video game by processing input provided by a user via an input device;
- as part of the video game, controlling at least one virtual object that is located within a virtual game space based on the input;
- rendering first image data that includes a view of the virtual games space by using a virtual camera, wherein the first image data is generated at a first resolution;
- arranging a plurality of first matrices that each include a plurality of values that are based on pixel values from a corresponding pixel group of a plurality of different pixel groups across the first image data;
- transforming, through using multiple layers of a neural network, the plurality of first matrices into a plurality of output matrices by using, for each of the multiple layers, input data with left matrix multiplication to a left learned matrix and a right matrix multiplication and a right learned matrix;
- and
- outputting, to the display device, an upscaled image generated based on values from the plurality of output matrices generated by use of the neural network values, wherein the upscaled image is output in real-time with the rendering of the first image data.
A computer program product stored to a non-transitory storage medium, the computer program product for execution by a processing system that includes a central processing unit (CPU) and a graphical processing unit (GPU), the GPU including a plurality of processing cores, the computer program product comprising instructions that, when executed, are configured to cause the processing system to perform operations comprising:
- executing a video game by processing input provided by a user via an input device;
- as part of the video game, controlling at least one virtual object that is located within a virtual game space based on the input;
- rendering first image data that includes a view of the virtual games space by using a virtual camera, wherein the first image data is generated at a first resolution;
- arranging a plurality of first matrices that each include a plurality of values that are based on pixel values from a corresponding pixel group of a plurality of different pixel groups across the first image data;
- transforming, through using multiple layers of a neural network, the plurality of first matrices into a plurality of output matrices by using, for each of the multiple layers, input data with a left matrix multiplication and a left learned matrix and a right matrix multiplication and a right learned matrix;
- and
- outputting, to a display device that is coupled to the processing system, an upscaled image generated based on values from the plurality of output matrices generated by use of the neural network values, wherein the upscaled image is output in real-time with the rendering of the first image data.
A method of upconverting rendered image data in real-time for display on a display device, the method comprising:
- executing, on a processing system that includes a central processing unit (CPU) and a graphical processing unit (GPU), the GPU including a plurality of processing cores, a video game by processing input provided by a user via an input device;
- as part of the video game, controlling at least one virtual object that is located within a virtual game space based on the input;
- rendering first image data that includes a view of the virtual games space by using a virtual camera, wherein the first image data is generated at a first resolution;
- arranging a plurality of first matrices that each include a plurality of values that are based on pixel values from a corresponding pixel group of a plurality of different pixel groups across the first image data;
- transforming, through using multiple layers of a neural network, the plurality of first matrices into a plurality of output matrices by using, for each of the multiple layers, input data with a left matrix multiplication and a left learned matrix and a right matrix multiplication and a right learned matrix;
- and
- outputting, to the display device, an upscaled image generated based on values from the plurality of output matrices generated by use of the neural network values, wherein the upscaled image is output in real-time with the rendering of the first image data.
Dependencies are inferred from the claim text.