Parallel processing of a Softmax operation by dividing an input vector into a plurality of fragments

A computer program embodied on non-transitory computer-readable storage, the program comprising code configured so as when run on a hardware processor comprising a plurality of processing units on a single chip, each processing unit comprising an execution unit and having access to computer memory, wherein each processing unit is associated with and has access to its own computer memory which is not shared by others of the processing units, wherein the computer memory of each processing unit stores the code executable by the execution unit of that processing unit, wherein an input vector is divided into a plurality of fragments, wherein there are no dependencies among any of the plurality of fragments, and wherein each processing unit further comprises a respective plurality of registers, the hardware processor performs a method of executing a neural network by processing the input vector comprising a plurality of input values by performing operations including:

• storing in each computer memory a respective fragment of the input vector;
• the execution units operating in parallel wherein each execution unit accesses its own computer memory in a selecting step to obtain input values from its fragment of the input vector;
• computing an activation function of the input vector to generate an output vector, the output vector comprising one or more output values, wherein the computing the activation function of the input vector to generate the output vector comprises:
• each execution unit selecting, in the selecting step, from the input values of its fragment a largest value, wherein the largest value is determined by reading successive pairs of input values of its fragment into the respective plurality of registers and recursively determining a maximum value of pairs of values stored in the registers, wherein the largest value is determined to be an approximation of a natural logarithm of a sum of an exponential of each of the input values;
• sharing the largest value from the input values of each fragment with the other processing units and obtaining a maximum result value based on the largest value of all fragments and storing the maximum result value in the computer memory of each processing unit;
• the execution units operating in parallel wherein each execution unit accesses its computer memory in a computing step to obtain each input value of its fragment and the maximum result value;
• and
• computing each output value for the output vector, the computing comprising subtracting the maximum result value from each respective input value of the respective fragment to generate a respective output value of the respective fragment based on a natural logarithm of the activation function and combining the respective output values of all the fragments to generate the output vector,
• wherein the neural network comprises a layer, and wherein the input vector is an input to the layer and the output vector is an output of the layer.

A computer configured to execute a neural network comprising a layer, the computer comprising:

• a plurality of processing units, each processing unit comprising an execution unit and having access to computer memory, wherein each processing unit is associated with and has access to its own computer memory which is not shared by others of the processing units, wherein the computer memory stores code executable by the execution unit and input values of an input vector to be processed by the code, wherein the input vector is divided into a plurality of fragments, wherein there are no dependencies among any of the plurality of fragments, and wherein each processing unit further comprises a respective plurality of registers, wherein the computer memory of each processing unit stores a set of input values of the input vector constituting a respective fragment of the plurality of fragments, and wherein the code, when executed, is configured to cause the processing units to operate in parallel wherein each processing unit is configured to:
• process its fragment by accessing its associated computer memory to obtain multiple pairs of the input values of its fragment of the input vector and reading the multiple pairs into the respective plurality of registers;
• recursively determine a maximum or a corrected maximum input value of each pair stored in the registers as a maximum result element of its fragment;
• store in the computer memory a maximum or a corrected maximum result of the maximum result element of the multiple pairs as an approximation to a natural log of a sum of exponents of the input values of its fragment,
• share its maximum or corrected maximum result with other processing units which are configured to process other fragments of the input vector to generate respective other maximum or corrected maximum results, the code configured to cause each processing unit to combine its maximum or corrected maximum result with the respective other maximum or corrected maximum results received from the other processing units to generate a maximum or corrected maximum result of the input vector;
• and
• access its computer memory to obtain each input value of its fragment and apply each input value to the maximum or the corrected maximum result of the input vector to generate each output value of an output fragment of a Softmax output vector,
• wherein the input vector is an input to the layer of the neural network, and wherein the output vector comprises the output fragments and is an output of the layer.