Hiding latency of multiplier-accumulator using partial results

Computational apparatus, comprising:

• a memory, which is configured to contain multiple matrices of input data values;
• an array of processing elements, each configured to perform multiplications of respective first and second input operands and to accumulate products of the multiplication to generate respective output values;
• and
• data access logic, which is configured to select from the memory a plurality of mutually-disjoint first matrices and a second matrix, and to distribute to the processing elements the input data values in a sequence that is interleaved among the first matrices, along with corresponding input data values from the second matrix, so as to cause the processing elements to compute, in the interleaved sequence, respective matrix multiplications of each of the first matrices with the second matrix,
• wherein the data access logic is configured to transpose at least one of the first and second matrices before distributing the input data values.

A method for computation, comprising:

• storing in a memory multiple matrices of input data values;
• distributing the input data values from the first matrices in a sequence that is interleaved among the first matrices, along with corresponding input data values from the second matrix, to an array of processing elements, each configured to perform multiplications of respective first and second input operands and to accumulate products of the multiplication to generate respective output values;
• and
• multiplying the distributed data values and accumulating the products in the processing elements so as to compute, in the interleaved sequence, respective matrix multiplications of the first matrices with the second matrix,
• wherein the processing elements have a latency of a given number of clock cycles for each multiply-accumulate operation, and wherein distributing the input data values comprises selecting and distributing the input data values in the interleaved sequence from a number of the first matrices that is equal to the given number.

A method for computation, comprising:

• storing in a memory multiple matrices of input data values;
• distributing the input data values from the first matrices in a sequence that is interleaved among the first matrices, along with corresponding input data values from the second matrix, to an array of processing elements, each configured to perform multiplications of respective first and second input operands and to accumulate products of the multiplication to generate respective output values;
• and
• multiplying the distributed data values and accumulating the products in the processing elements so as to compute, in the interleaved sequence, respective matrix multiplications of the first matrices with the second matrix,
• wherein distributing the input data values comprises transposing at least one of the first and second matrices before distributing the input data values.