Pruning and accelerating neural networks with hierarchical fine-grained structured sparsity

A system, comprising:

• a memory that stores a first array;
• and
• a processor that is connected to the memory, wherein the processor comprises a plurality of multi-element processing units and is configured to:
• prune initial parameters of the first array for a dense neural network model according to a first level of a hierarchical structured sparsity pattern to produce pruned parameters for a single-level structured sparse neural network model corresponding to a first sparsity degree;
• prune the pruned parameters according to a second level to produce first parameters of the first array for a first hierarchical structured sparse neural network model corresponding to a second sparsity degree that is different compared with the first sparsity degree;
• prune the pruned parameters according to the second level to produce second pruned parameters for a second hierarchical structured sparse neural network model corresponding to a third sparsity degree that is different compared with the second sparsity degree;
• receive the first parameters, wherein the hierarchical structural sparsity pattern includes at least a per-element pattern corresponding to the first level that indicates positions of the first parameters within the first array and a multi-element pattern corresponding to the second level and that identifies positions of multi-elements within the first array;
• for each result of an output array, select and process a subset of second parameters of a second array and a corresponding subset of the first parameters according to the per-element pattern to produce intermediate results comprising sums of products;
• and
• based on the multi-element pattern, combine the intermediate results to compute a subset of the results of the output array, wherein circuitry comprising at least a first multi-element processing unit of the plurality of multi-element processing units is selectively disabled based on the multi-element pattern, circuitry comprising at least one and less than all processing elements within a second multi-element processing unit of the plurality of multi-element processing units is selectively disabled based on the per-element pattern, and transmission of the subset of second parameters to the first multi-element processing unit is prevented.

A non-transitory computer-readable media storing computer instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of:

• pruning initial parameters of a first array for a dense neural network model according to a first level of a hierarchical structured sparsity pattern to produce pruned parameters for a single-level structured sparse neural network model corresponding to a first sparsity degree;
• pruning the pruned parameters according to a second level to produce first parameters of the first array for a first hierarchical structured sparse neural network model corresponding to a second sparsity degree that is different compared with the first sparsity degree;
• pruning the pruned parameters according to the second level to produce second pruned parameters for a second hierarchical structured sparse neural network model corresponding to a third sparsity degree that is different compared with the second sparsity degree;
• receiving the first parameters, wherein the hierarchical structural sparsity pattern includes at least a per-element pattern corresponding to the first level that indicates positions of the first parameters within the first array and a multi-element pattern corresponding to the second level and that identifies positions of multi-elements within the first array;
• for each result of an output array, selecting and processing a subset of second parameters of a second array and a corresponding subset of the first parameters according to the per-element pattern to produce intermediate results comprising sums of products;
• and
• based on the multi-element pattern, combining the intermediate results to compute a subset of the results of the output array, wherein the combining comprises selectively disabling circuitry comprising at least a first multi-element processing unit of a plurality of multi-element processing units based on the multi-element pattern, selectively-disabling circuitry comprising at least one and less than all processing elements within a second multi-element processing unit of the plurality of multi-element processing units based on the per-element pattern, and preventing transmission of the subset of second parameters to the first multi-element processing unit.