Parallelizing the training of convolutional neural networks

A method for training an image processing convolutional neural network on a plurality of batches of training examples, wherein the image processing convolutional neural network is configured to receive as input one or more images and wherein each of the training examples comprises one or more images, the image processing convolutional neural network having a plurality of layers arranged in a sequence from lowest to highest, the sequence including one or more convolutional layers followed by one or more fully-connected layers, each convolutional layer and each fully-connected layer comprising a respective plurality of nodes, the method comprising:

• maintaining, by each of a plurality of workers that are implemented by one or more computers, a respective replica of each of the convolutional layers, wherein each replica of a convolutional layer includes all of the nodes in the convolutional layer;
• maintaining, by each of the workers, a respective disjoint partition of each of the fully-connected layers, wherein each disjoint partition of a fully-connected layer includes a portion of the nodes of the fully-connected layer;
• receiving, by each of the workers, a batch of training examples assigned to the worker, wherein the batches of training examples are assigned such that each worker receives a respective batch of the plurality of batches;
• training the convolutional layer replica maintained by the worker on the batch of training examples assigned to the worker, comprising:
• processing the batch of training examples through all of the convolutional layer replicas maintained by the worker to compute respective convolutional activations for each training example in the batch assigned to the worker;
• and
• training the fully-connected layer partitions maintained by the worker on each of the plurality of batches of training examples, comprising:
• sending own convolutional data to other workers in the plurality of workers, the own convolutional data comprising the respective convolutional activations for training examples assigned to the worker,
• receiving other convolutional data from other workers in the plurality of workers, the other convolutional data comprising respective convolutional activations for other training examples assigned to other workers, and
• in parallel with receiving the other convolutional data, processing the own convolutional data through a partition of a lowest fully-connected layer maintained by the worker to generate own initial fully-connected layer partition activations for the corresponding training examples.

One or more non-transitory computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training an image processing convolutional neural network on a plurality of batches of training examples, wherein the image processing convolutional neural network is configured to receive as input one or more images and wherein each training example comprises one or more images, the image processing convolutional neural network having a plurality of layers arranged in a sequence from lowest to highest, the sequence including one or more convolutional layers followed by one or more fully-connected layers, each convolutional layer and each fully-connected layer comprising a respective plurality of nodes, the operations comprising:

• maintaining, by each of a plurality of workers that are implemented by one or more computers, a respective replica of each of the convolutional layers, wherein each replica of a convolutional layer includes all of the nodes in the convolutional layer;
• maintaining, by each of the workers, a respective disjoint partition of each of the fully-connected layers, wherein each disjoint partition of a fully-connected layer includes a portion of the nodes of the fully-connected layer;
• receiving, by each of the workers, a batch of training examples assigned to the worker, wherein the batches of training examples are assigned such that each worker receives a respective batch of the plurality of batches;
• training the convolutional layer replica maintained by the worker on the batch of training examples assigned to the worker, comprising:
• processing the batch of training examples through all of the convolutional layer replicas maintained by the worker to compute respective convolutional activations for each training example in the batch assigned to the worker;
• and
• training the fully-connected layer partitions maintained by the worker on each of the plurality of batches of training examples, comprising:
• sending own convolutional data to other workers in the plurality of workers, the own convolutional data comprising the respective convolutional activations for training examples assigned to the worker,
• receiving other convolutional data from other workers in the plurality of workers, the other convolutional data comprising respective convolutional activations for other training examples assigned to other workers, and
• in parallel with receiving the other convolutional data, processing the own convolutional data through a partition of a lowest fully-connected layer maintained by the worker to generate own initial fully-connected layer partition activations for the corresponding training examples.