Apparatus and method of performing matrix multiplication operation of neural network

A method in a neural network apparatus, the method comprising:

• accessing at least a portion of a neural network comprising an input layer, multiple hidden layers, and an output layer, each layer comprising a respective set of nodes, each layer other than the input layer having weights of connections to a preceding adjacent layer, the neural network further comprising a weight matrix and a number of heads of the neural network, the weight matrix comprising weight values of the weights, the weight matrix having a row dimension and a column dimension
• reading an input feature map and the weight matrix from a memory, wherein a matrix operation is performed on the input feature map and the weight matrix, wherein the matrix operation comprises a matrix multiplication operation, a transpose operation, and a reshape operation, wherein the method is capable of performing the transpose operation and the reshape operation before the matrix multiplication operation, and wherein the method is capable of performing the transpose operation and the reshape operation after the matrix multiplication operation;
• selecting between a column dimension and a row dimension as a split dimension, wherein the split dimension is selected according to whether the reshape operation and the transpose operation are to be performed before or after the matrix multiplication operation;
• splitting, in the selected split dimension, the weight matrix into a number weight sub-matrices based on the number of heads;
• generating intermediate feature maps by performing the matrix multiplication operation between the input feature map and the weight sub-matrices;
• generating a final feature map by performing an operation on the intermediate feature maps;
• and
• generating activations of nodes, according to an activation function, of one of the layers of the neural network based on the final feature map.

A method performed by one or more processors, the method comprising:

• reading from a memory an input feature map and at least a portion of a neural network comprising an input layer, multiple hidden layers, and an output layer, each layer comprising a respective set of nodes, each layer other than the input layer having weights of connections to a preceding adjacent layer, the neural network further comprising a weight matrix, the weight matrix comprising weight values of the weights, the weight matrix having a row dimension and a column dimension, wherein a matrix operation is to be performed on the input feature map and the weight matrix, wherein the matrix operation comprises a matrix multiplication operation, and wherein the method is capable of performing the transpose operation and the reshape operation before the matrix multiplication operation, and wherein the method is capable of performing the transpose operation and the reshape operation after the matrix multiplication operation;
• determining whether a column split dimension is a column dimension or a row dimension based on whether the transpose and reshape operations are performed before or after the matrix multiplication operation;
• splitting the weight matrix into weight sub-matrices based on the number of heads;
• performing the matrix multiplication operation between the input feature map and each of the weight sub-matrices to generate intermediate feature maps respectively corresponding to the weight sub-matrices, wherein an intermediate feature map is generated based on activations of nodes of a given layer according an intermediate feature map outputted by a layer preceding the given layer;
• and
• performing an operation on the intermediate feature maps to generate an output feature map, and storing the output feature map in the memory.