Neural network pruning with cyclical sparsity

A computing device, comprising a processing device configured to perform operations comprising:

• receiving as an input a weight tensor for a neural network and a sparsity parameter associated with a rate of adjusting sparsity of the weight tensor, wherein the rate is based on a training cycle of the neural network;
• increasing, based on the sparsity parameter, a level of sparsity of the weight tensor at the rate associated with the training cycle to generate a sparse weight tensor;
• updating the neural network using the sparse weight tensor generating an updated weight tensor;
• decreasing, based on the sparsity parameter, a level of sparsity of the updated weight tensor at the rate associated with the training cycle to generate a dense weight tensor;
• increasing, based on the sparsity parameter, the level of sparsity of the dense weight tensor at the rate associated with the training cycle to generate a final sparse weight tensor;
• and
• using the neural network with the final sparse weight tensor to generate inferences.

A computing device, comprising:

• means for receiving as an input a weight tensor for a neural network and a sparsity parameter associated with a rate of adjusting sparsity of the weight tensor, wherein the rate is based on a training cycle of the neural network;
• means for increasing, based on the sparsity parameter, a level of sparsity of the weight tensor at the rate associated with the training cycle to generate a sparse weight tensor;
• means for updating the neural network using the sparse weight tensor generating an updated weight tensor;
• means for decreasing, based on the sparsity parameter, a level of sparsity of the updated weight tensor at the rate associated with the training cycle to generate a dense weight tensor;
• means for increasing, based on the sparsity parameter, the level of sparsity of the dense weight tensor at the rate associated with the training cycle to generate a final sparse weight tensor;
• and
• means for using the neural network with the final sparse weight tensor to generate inferences.

A non-transitory processor-readable medium having stored thereon processor-executable instructions configured to cause a processing device to perform operations comprising:

• receiving as an input a weight tensor for a neural network and a sparsity parameter associated with a rate of adjusting sparsity of the weight tensor, wherein the rate is based on a training cycle of the neural network;
• increasing, based on the sparsity parameter, a level of sparsity of the weight tensor at the rate associated with the training cycle to generate a sparse weight tensor;
• updating the neural network using the sparse weight tensor to generate an updated weight tensor;
• decreasing, based on the sparsity parameter, a level of sparsity of the updated weight tensor at the rate associated with the training cycle to generate a dense weight tensor;
• increasing, based on the sparsity parameter, the level of sparsity of the dense weight tensor at the rate associated with the training cycle to generate a final sparse weight tensor;
• and
• using the neural network with the final sparse weight tensor to generate inferences.