Parallel execution of gated activation unit operations

The method of any preceding claim, wherein the gated activation unit is a gated recurrent unit or a long short term memory unit of the neural network.

The method of any preceding claim, wherein generating, by each thread block, a portion of the current state vector comprises generating, by a plurality of thread blocks, multiple portions of the current state vector at least partially in parallel.

The method of any preceding claim, wherein the operations of a layer comprise a bias addition operation between a result of a matrix operation for the layer and a bias vector for the layer, and further comprising:

• distributing portions of the bias vector to each thread block of the plurality of thread blocks;
• and
• adding, by each thread block, a portion of the bias vector to a result of a matrix operation.

The method of any preceding claim, further comprising interleaving rows of the plurality of weight matrices by warps such that all warps receive some rows from every weight matrix and corresponding rows from every weight matrix are assigned to the same warp.

The method of any preceding claim, further comprising interleaving values of the plurality of weight matrices by threads such that all threads receive some values from every weight matrix and corresponding values from every weight matrix are assigned to the same thread.

A system configured to implement a neural network on a parallel processing device, the neural network comprising:

• at least one gated activation unit having a plurality of weight matrices, the gated activation unit having two or more layers, each layer defining operations comprising:
• (i) a matrix operation between a weight matrix for the layer and concatenated input vectors and (ii) a nonlinear activation operation using a result of the matrix operation;
• wherein the system is configured to interleave rows of the plurality of weight matrices by assigning groups of corresponding rows to respective thread blocks, each thread block being a computation unit for execution by an independent processing unit of a plurality of independent processing units of the parallel processing device.