Training deep neural networks with 4-bit integers

A computer-implemented method for accelerating operations of a deep neural network, comprising:

• for each of a plurality of matrix multiplications during forward propagation, obtaining a first matrix by multiplying an activation matrix, which is an input of the matrix multiplication, with a block-diagonal Hadamard transform matrix;
• obtaining a second matrix by multiplying a weight matrix, a transpose of which is another input of the matrix multiplication, with the block-diagonal Hadamard transform matrix;
• quantizing the first matrix and the second matrix into 4-bit integer matrices;
• obtaining a resultant matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix;
• and
• dequantizing the resultant matrix to a matrix of floating-point representation.

A computing device, comprising:

• memory for storing instructions;
• and
• one or more processing units coupled to the memory, wherein the instructions, when executed by the one or more processing units, cause the one or more processing units to perform a method for accelerating operations of a deep neural network, the method comprising:
• for each of a plurality of matrix multiplications during forward propagation,
• obtaining a first matrix by multiplying an activation matrix, which is an input of the matrix multiplication, with a block-diagonal Hadamard transform matrix;
• obtaining a second matrix by multiplying a weight matrix, a transpose of which is another input of the matrix multiplication, with the block-diagonal Hadamard transform matrix;
• quantizing the first matrix and the second matrix into 4-bit integer matrices;
• obtaining a resultant matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix;
• and
• dequantizing the resultant matrix to a matrix of floating-point representation.

A non-transitory computer-readable storage medium on which are stored instructions for accelerating operations of a deep neural network, the instructions, when executed by one or more processors, causing the one or more processors to perform the following steps:

• for each of a plurality of matrix multiplications during forward propagation, obtaining a first matrix by multiplying an activation matrix, which is an input of the matrix multiplication, with a block-diagonal Hadamard transform matrix;
• obtaining a second matrix by multiplying a weight matrix, a transpose of which is another input of the matrix multiplication, with the block-diagonal Hadamard transform matrix;
• quantizing the first matrix and the second matrix into 4-bit integer matrices;
• obtaining a resultant matrix by multiplying the quantized first matrix and the transpose of the quantized second matrix;
• and
• dequantizing the resultant matrix to a matrix of floating-point representation.