Training convolutional neural networks on graphics processing units
AssigneeMicrosoft Corporation
InventorSiddhartha Puri
Life of this patent
Estimating the term…
Abstract
A convolutional neural network is implemented on a graphics processing unit. The network is then trained through a series of forward and backward passes, with convolutional kernels and bias matrices modified on each backward pass according to a gradient of an error function. The implementation takes advantage of parallel processing capabilities of pixel shader units on a GPU, and utilizes a set of start-to-finish formulas to program the computations on the pixel shaders. Input and output to the program is done through textures, and a multi-pass summation process is used when sums are needed across pixel shader unit registers.
Claims18
A computer-implemented method for training a convolutional neural network to identify images using graphics data which can be read by a graphics processing unit (“GPU”), and one or more GPU-executable programs, the method comprising:
- receiving the graphics data representing a state of the convolutional neural network and comprising one or more textures representing one or more neural network variables, wherein the one or more textures comprises a texture with two-dimensional addressing, and at least one or more of the textures represents a neural network variable with addressing of more than two dimensions which has been flattened into two dimensional addressing, the convolutional neural network comprising at least one layer comprising a plurality of patches;
- executing one or more of the GPU-executable programs on the GPU in order to perform a forward pass in the convolutional neural network, the executing including performing convolution operations on the patches;
- executing one or more of the GPU-executable programs on the GPU in order to perform a backward pass in the convolutional neural network, the executing including performing convolution operations on the patches;
- executing one or more of the GPU-executable programs on the GPU in order to modify the patches in the convolutional neural network by changing the graphics data based on results of the backward pass;
- and
- repeating executing one or more of the GPU-executable programs to perform forward passes, backward passes, and to modify the graphics data until the convolutional neural network is trained.
One or more computer-readable storage media storing instructions which, when executed on a graphics card, cause the graphics card to perform a method for training a convolutional neural network, the method comprising:
- receiving a plurality of textures, the plurality of textures at least in part representing square convolutional kernels for the neural network, wherein at least some of the textures have two-dimensional addressing, and represent square convolutional kernels with addressing of more than two dimensions which have been flattened into two dimensional addressing;
- computing a plurality of forward passes of the neural network on a plurality of input data including convoluting and subsampling the square convolutional kernels;
- for each of the plurality of forward passes, computing a backward pass of the neural network using a gradient function;
- and
- for each backward pass, based on the results of the gradient function, changing information contained in the square convolutional kernels from the plurality of textures to affect a training of the neural network.
A graphics processing unit configured to perform a method for training a handwriting-recognition convolutional neural network, wherein the convolutional neural network comprises one or more of layers, at least some of the layers each comprising a plurality of square convolutional kernel patches, and wherein the graphics processing unit comprises:
- data storage, configured to store one or more graphics textures, the graphics textures describing the square convolutional kernel patches of the handwriting-recognition neural network, wherein at least some of the graphical textures have two-dimensional addressing and represent square convolutional kernel patches having addressing of more than two dimensions which have been flattened into two dimensional addressing;
- a plurality of pixel shader units configured via pixel shader programming:
- to perform repeated forward passes and backward passes of the neural network on handwriting input data, the passes including performing convolutional operations on the square convolutional kernel patches;
- to store results in the plurality of graphics textures;
- and
- to modify the square convolutional kernel patches of the plurality of textures based on results of the forward and backward passes in order to train the neural network.
Dependencies are inferred from the claim text.