Method and apparatus for reducing the parameter density of a deep neural network (DNN)

A method comprising:

• performing layer-wise pruning a specified set of parameters from each layer of a reference neural network model in accordance with a Joint Feed-forward and Backward Propagation Approximation (JFBPA) to generate a second neural network model having a higher sparsity rate than the reference neural network model, wherein a layer in the second neural network model is computed using the JFBPA based on a number of zero parameters in a corresponding matrix;
• retraining the second neural network model in accordance with a set of training data to generate a retrained second neural network model;
• and
• designating the retrained second neural network model as a final neural network model if a specified target sparsity rate or a specified number of alternating pruning and retraining iterations has been reached, and performing additional layer-wise pruning of the retrained second neural network model if both the specified target sparsity rate and the specified number of alternating pruning and retraining iterations have not been reached.

A non-transitory machine-readable medium having program code stored thereon which, when executed by a machine, causes the machine to perform:

• performing layer-wise pruning a specified set of parameters from each layer of a reference neural network model in accordance with a Joint Feed-forward and Backward Propagation Approximation (JFBPA) to generate a second neural network model having a higher sparsity rate than the reference neural network model, wherein a layer in the second neural network model is computed using the JFBPA based on a number of zero parameters in a corresponding matrix;
• retraining the second neural network model in accordance with a set of training data to generate a retrained second neural network model, wherein the retraining of the layer in the second neural network model maintains the number of zero parameters;
• and
• designating the retrained second neural network model as a final neural network model if a specified target sparsity rate or a specified number of alternating pruning and retraining iterations has been reached, and performing additional layer-wise pruning of the retrained second neural network model if both the specified target sparsity rate and the specified number of alternating pruning and retraining iterations have not been reached.