Back to results
US 11,961,001 B2Granted
Parallel forward and backward propagation
AssigneeNVIDIA Corporation
InventorMaxim Andreyevich Naumov
Life of this patent
Estimating the term…
Abstract
A neural network structure is separated into an odd neural network including only the odd layers and an even neural network including only the even layers. In order to allow for parallel execution, for forward propagation a second input is generated from the original input, while for backward propagation a second error gradient is generated. Parallel execution may accelerate the forward and backward propagation operations without significant change in accuracy of the model. Additionally, restructuring a single neural network into two or more parallel neural networks may reduce the total time needed for training.
Claims20
A computer-implemented method, comprising:
- identifying even processing layers and odd processing layers in a sequence of multiple processing layers included in a neural network model, wherein the sequence alternates between each of the odd processing layers and each of the even processing layers;
- decoupling the odd processing layers and the even processing layers to produce a parallel neural network system comprising an even neural network model that includes the even processing layers in a first sequence and does not include the odd processing layers and a separate and independent odd neural network model that is decoupled from the even neural network model and includes the odd processing layers in a second sequence and does not include the even processing layers;
- processing, based on a block bi-diagonal lower triangular system of equations, odd input data by the odd neural network model according to odd parameter values to produce an odd output and even input data by the even neural network model according to even parameter values to produce an even output;
- updating the odd parameter values based on a first difference between the odd output and an odd ground truth output;
- and
- updating the even parameter values based on a second difference between the even output and an even ground truth output.
3 more elements
A system, comprising:
- a memory that stores an even ground truth output and an odd ground truth output;
- and
- a processor that is connected to the memory, wherein the processor is configured to train a neural network model by:
- identifying even processing layers and odd processing layers in a sequence of multiple processing layers included in a neural network model into odd processing layers and even processing layers, wherein the sequence alternates between each of the odd processing layers and each of the even processing layers;
- decoupling the odd processing layers and the even processing layers to produce a parallel neural network system comprising an even neural network model that includes the even processing layers in a first sequence and does not include the odd processing layers and a separate and independent odd neural network model that is decoupled from the even neural network model and includes the odd processing layers in a second sequence and does not include the even processing layers;
- processing, based on a block bi-diagonal lower triangular system of equations, odd input data by the odd neural network model according to odd parameter values to produce an odd output and processing even input data by the even neural network model according to even parameter values to produce an even output;
- updating the odd parameter values based on a first difference between the odd output and an odd ground truth output;
- and
- updating the even parameter values based on a second difference between the even output and an even ground truth output.
6 more elements
A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:
- identifying even processing layers and odd processing layers in a sequence of multiple processing layers included in a neural network model into odd processing layers and even processing layers, wherein the sequence alternates between each of the odd processing layers and each of the even processing layers;
- decoupling the odd processing layers and the even processing layers to produce a parallel neural network system comprising an even neural network model that includes the even processing layers in a first sequence and does not include the odd processing layers and a separate and independent odd neural network model that is decoupled from the even neural network model and includes the odd processing layers in a second sequence and does not include the even processing layers;
- processing, based on a block bi-diagonal lower triangular system of equations, odd input data by the odd neural network model according to odd parameter values to produce an odd output and processing even input data by the even neural network model according to even parameter values to produce an even output;
- updating the odd parameter values based on a first difference between the odd output and an odd ground truth output;
- and
- updating the even parameter values based on a second difference between the even output and an even ground truth output.
3 more elements
Dependencies are inferred from the claim text.