Efficient parallel training of a network model on multiple graphics processing units

A computer-implemented method executed by at least one central processing unit (CPU) for training a neural network model comprising:

• collecting gradients of each layer of a plurality of layers during backpropagation of training sent from a plurality of graphics processing units (GPUs);
• accumulating the gradients of each layer on the CPU concurrently while the GPUs continue backpropagation of a subsequent layer to overlap backpropagation, gradient transfer and accumulation;
• and
• delivering the gradients of each layer during the backpropagation of a subsequent layer from the CPU to GPUs after summing up all gradients of each layer.

A computer-implemented method executed by at least one central processing unit (CPU) for training a neural network model comprising:

• collecting gradients of each layer of a plurality of layers during backpropagation of training sent from a plurality of graphics processing units (GPUs);
• accumulating the gradients of each layer on the CPU concurrently while the GPUs continue backpropagation execution of subsequent layers to overlap backpropagation, gradient transfer and accumulation;
• storing partial gradients of each layer collected from the GPUs in an array in a memory coupled to the CPU and further storing accumulated gradients of each layer collected from the GPUs in the array;
• creating a communication stream to send the gradients of each layer to a host for accumulation and delivering the gradients of each layer during the backpropagation from the CPU to GPUs after summing up all gradients of each layer;
• wherein at least one of the GPUs further updates whole model parameters of the neural network model after the receipt of a gradient of a last layer;
• and
• wherein the backpropagation is part of a synchronous stochastic gradient descent algorithm for training the neural network model on a convolutional neural network (CNN).