Learning a truncation rank of singular value decomposed matrices representing weight tensors in neural networks

An apparatus of learning a rank of an artificial neural network, comprising:

• a memory;
• and
• at least one processor coupled to the memory, the at least one processor being configured to:
• decompose a weight tensor into a first weight tensor and a second weight tensor, the weight tensor comprising a plurality of weights upon which activations of artificial neurons of the artificial neural network are dependent;
• apply a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the rank of the first weight tensor and the second weight tensor, the set of rank selection parameters being initialized to be monotonically decreasing in order of singular value magnitude;
• and
• update the set of rank selection parameters by averaging updates calculated for each rank selection parameter of the set of rank selection parameters.

An apparatus for operating an artificial neural network, comprising:

• means for decomposing a weight tensor into a first weight tensor and a second weight tensor, the weight tensor comprising a plurality of weights upon which activations of artificial neurons of the artificial neural network are dependent;
• means for applying a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the rank of the first weight tensor and the second weight tensor, the set of rank selection parameters being initialized to be monotonically decreasing in order of singular value magnitude;
• and
• means for updating the set of rank selection parameters, the set of rank selection parameters being updated based on updates calculated for each rank selection parameter.

A non-transitory, computer-readable medium storing executable code for learning a rank of an artificial neural network, comprising code to:

• decompose a weight tensor into a first weight tensor and a second weight tensor, the weight tensor comprising a plurality of weights upon which activations of artificial neurons of the artificial neural network are dependent;
• apply a set of rank selection parameters to the first weight tensor and the second weight tensor to truncate the rank of the first weight tensor and the second weight tensor, the set of rank selection parameters being initialized to be monotonically decreasing in order of singular value magnitude;
• and
• update the set of rank selection parameters by averaging updates calculated for each rank selection parameter of the set of rank selection parameters.