Back to results
US 11,379,951 B2Granted
Systems and methods for machine learned image conversion
AssigneeNintendo Co., Ltd.
InventorsAlexandre Delattre, Théo Charvet, Raphaël Poncet
Life of this patent
Estimating the term…
Abstract
A computer system is provided for converting images through use of a trained neural network. A source image is divided into blocks and context data is added to each pixel block. The context blocks are split into channels and each channel from the same context block is added to the same activation matrix. The action matrix is then executed against a trained neural network to produce a changed activation matrix. The changed activation matrix is then used to generate a converted image.
Claims20
A method of transforming signal data using a neural network, the method comprising:
- (a) populating an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- (b) applying, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers and wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- (c) outputting the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein each of the plurality of samples is a plurality of pixels from a source image, which is the source signal, that is in a first resolution and the transformed signal is an image in a second resolution.
2 more elements
A method of transforming signal data using a neural network, the method comprising:
- (a) populating an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- (b) applying, over multiple layers of the neural network, separable block transforms that are based on at least at first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers and wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- (c) outputting the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein at least two of the rows or columns of the initial activation matrix correspond to superposable data from each of the plurality of samples.
2 more elements
A method of transforming signal data using a neural network, the method comprising:
- (a) populating an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- (b) applying, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers are wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- (c) outputting the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein the first and second learned matrices are a sample-wise matrix and a channel-wise matrix,
- wherein the sample-wise matrix applies a transform to all channel values of each activation sample within the input activation matrix independently from the sample position.
3 more elements
A method of transforming signal data using a neural network, the method comprising:
- (a) populating an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- (b) applying, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers and wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- (c) outputting the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein the first and second learned matrices are a sample-wise matrix and a channel-wise matrix,
- wherein the channel-wise matrix apples a transform to all sample values of each activation channel independently from the channel position.
3 more elements
A method of transforming signal data using a neural network, the method comprising:
- (a) populating an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- (b) applying, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers and wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- (c) outputting the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein the first matrix is multiplied on the left of the input activation matrix and the second matrix is multiplied on the right of the input activation matrix.
2 more elements
A computer program product stored to a non-transitory storage medium, the computer program product for execution by a processing system that includes at least one hardware processor, the computer program product comprising instructions that, when executed, cause the processing system to:
- construct an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- apply, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers and wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- output the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein the source signal in a source image and the transformed signal is a transformed image,
- wherein the source image is at a first resolution and the transformed image is at a second resolution that is different from the first resolution.
3 more elements
A computer system comprising:
- a processing system that includes at least one hardware processor, the processing system configured to:
- populate an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- apply, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix is used as the input activation matrix for a first layer of the multiple layers and wherein the input activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- output the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein the source signal is a source image and the transformed signal is a transformed image,
- wherein the source image is at a first resolution and the transformed image is at a second resolution that is different from the first resolution.
4 more elements
A computer system comprising:
- a processing system that includes at least one hardware processor, the processing system configured to:
- populate an initial activation matrix with a plurality of values that are based on data from a plurality of samples from a source signal;
- apply, over multiple layers of the neural network, separable block transforms that are based on at least a first learned matrix and a second learned matrix, to an input activation matrix to generate a corresponding output activation matrix, wherein the initial activation matrix for each successive layer is the output activation matrix of a prior layer;
- and
- output the output activation matrix of the last layer of the neural network to generate a transformed signal that is based on the output activation matrix of the last layer,
- wherein the source signal is generated based on execution of a computer application program and the transformed signal is output in connection with execution of the computer application program.
3 more elements
Dependencies are inferred from the claim text.