Reduction of parameters in fully connected layers of neural networks by low rank factorizations

A system for computing an output of a fully connected layer of a neural network, the fully connected layer comprising n inputs and m outputs the system comprising:

• a processing unit;
• and
• a non-transitory memory communicatively coupled to the processing unit and comprising computer-readable program instructions executable by the processing unit for:
• receiving a n-dimensional input vector x representative of the n inputs of the fully connected layer and a m-dimensional output vector y representative of the m outputs of the fully connected layer;
• partitioning the m-dimensional output vector y into equally sized subvectors yi, of length s and the n-dimensional input vector x into equally sized subvectors xj of length t, wherein s is a divisor of m and t is a divisor of n;
• generating matrices Uij∈Rr×t of rank r and matrices Vij∈Rs×r of rank r for i=(1, . . . , m/s) and j=(1 . . . , n/t), where r is a positive integer and r≤min(s, t) each of the matrices Uij comprising r×t connection parameters associated with logical connections between the n inputs and (m/s)*r*(n/t) logical intermediate nodes between the n inputs and the m outputs, and each of matrices Vij comprising the r×s connection parameters associated with logical connections between the (m/s)*r*(n/t) logical intermediate nodes and the m outputs;
• computing subvectors yi=Σj=1n/tVij(Uijxij) for i=(1, . . . , m/s) to reduce a number of multiplications by a factor of s×t/r(s+t);
• and
• computing the output vector y as the output of the fully connected layer of the neural network, the output vector computed by appending all the subvectors yi.

A computer-implemented method for computing an m-dimensional output vector y representative of m outputs of a fully connected layer of a neural network, the method comprising:

• receiving an n-dimensional input vector x representative of n inputs of the fully connected layer of the neural network;
• retrieving from memory matrices Uij ε Rr×t of rank r and matrices Vij ε Rs×r of rank r, each of the matrices Uij comprising r×t connection parameters associated with logical connections between the n inputs and (m/s)*r*(n/t) logical intermediate nodes between the n inputs and m outputs, and each of the matrices Vij comprising r×s connection parameters associated with logical connections between the (m/s)*r*(n/t) logical intermediate nodes and the m outputs, wherein s is a divisor of m and t is a divisor of n, r is a positive integer and r≤min(s, t);
• partitioning the input vector x into equally sized subvectors xj of length t;
• computing submatrices Wij as a product of matrices Uij and matrices Vij wherein a total number of connection parameters included in each submatrix Wij is r*(s+t);
• and
• computing the m-dimensional output vector y from y=W·x as the output of the fully connected layer of the neural network, where W is a matrix composed of all submatrices Wij for i=(1, . . . , m/s) and j=(1, . . . , n/t).

A computing system for computing an m-dimensional output vector y representative of m outputs of a fully connected layer of a neural network, the computing system comprising:

• a processing unit;
• and
• a non-transitory memory communicatively coupled to the processing unit and comprising computer-readable program instructions executable by the processing unit for:
• receiving an n-dimensional input vector x representative of n inputs of the fully connected layer of the neural network;
• retrieving from memory matrices Uij ε Rr×t of rank r and matrices Vij ε Rs×r of rank r, each of the matrices Uij comprising r×t connection parameters associated with logical connections between the n inputs and (m/s)*r*(n/t) logical intermediate nodes between the n inputs and m outputs, and each of the matrices Vij comprising r×s connection parameters associated with logical connections between the (m/s)*r*(n/t) logical intermediate nodes and the m outputs, wherein s is a divisor of m and t is a divisor of n, r is a positive integer and r≤min(s, t);
• partitioning the input vector x into equally sized subvectors xj of length t;
• computing submatrices Wij as a product of matrices Uij and matrices Vij wherein a total number of connection parameters included in each submatrix Wij is r*(s+t);
• and
• computing the m-dimensional output vector y from y=W·x, where W is a matrix composed of all submatrices Wij for i=(1, . . . , m/s) and j=(1, . . . , n/t).