US 10,482,380 B2 · Conditional parallel processing in fully-connected neural networks

Abstract

The present disclosure is directed to parallelization of artificial neural network processing by conditionally synchronizing, among multiple computer processors, either the input or output of individual operations, and by conditionally using either rows or columns of certain matrices used in the operations. The conditional processing may depend upon the relative sizes of the input and output of the specific operations to be performed. For example, if a current layer matrix of values is larger than a next layer matrix of values to be computed, then rows of a weight matrix may be used by the computer processors to compute the next layer matrix. If the current layer matrix is smaller than the next layer matrix, then columns of the weight matrix may be used by the computer processors to compute the next layer matrix.

A system comprising a plurality of processors, the system programmed by executable instructions to at least:

obtain data defining an artificial neural network, the artificial neural network comprising a first layer of nodes, a second layer of nodes, and a third layer of nodes, wherein the first layer comprises more nodes than the second layer, and wherein the second layer comprises fewer nodes than the third layer;
provide parallelization of processing operations of the artificial neural network to reduce communications between the plurality of processors by conditionally synchronizing, among the plurality of processors, individual processing operations of the plurality of processors, wherein the executable instructions to provide parallelization of the processing operations of the artificial neural network comprise executable instructions to:
provide to a first processor of the plurality of processors, based at least partly on the first layer comprising more nodes than the second layer and the second layer comprising fewer nodes than the third layer:
a first column of input data from a first data matrix, the first data matrix comprising input data for the artificial neural network;
a first row of weights from a first weight matrix, the first weight matrix comprising weights for connections between nodes of the first layer and nodes of the second layer;
and
a first column of weights from a second weight matrix, the second weight matrix comprising weights for connections between nodes of the second layer and nodes of the third layer;
provide to a second processor of the plurality of processors, based at least partly on the first layer comprising more nodes than the second layer and the second layer comprising fewer nodes than the third layer:
a second column of input data from the first data matrix;
a second row of weights from the first weight matrix;
and
a second column of weights from the second weight matrix;
compute, using the first processor, a first subset of columns of a second data matrix of values for the second layer, wherein the first subset is computed from the first column of input data, the first row of weights, and aggregated values received from the second processor of the plurality of processors;
compute, using the second processor, a second subset of columns of the second data matrix, wherein the second subset is computed from the second column of input data, the second row of weights, and aggregated values received from the first processor of the plurality of processors;
store, on each of the first and second processors, the second data matrix based at least partly on the second layer comprising fewer nodes than the third layer;
compute, using the first processor, a third subset of columns of a third data matrix of values for the third layer, wherein the third subset is computed from the second data matrix and the first column of weights;
compute, using the second processor, a fourth subset of columns of the third data matrix, wherein the fourth subset is computed from the second data matrix and the second column of weights;
and
generate, by the plurality of processors, output of the artificial neural network using the third data matrix.

16 more elements

Conditional parallel processing in fully-connected neural networks

Life of this patent

Conditional parallel processing in fully-connected neural networks

Life of this patent

Abstract

Claims20