Performing dynamic sparse computation on dense computation-efficient computing devices

A method for processing data in a Neural Network (NN) model comprising:

• identifying an operator in the NN model, the operator configured to perform a computation on at least one input tensor to generate an output tensor, the at least one input tensor storing input data along a plurality of dimensions in a sparse format;
• performing, during runtime, a permutation to rearrange the input data from the sparse format to a dense format, wherein the dense format is shorter than the sparse format along a dimension of the plurality of dimensions;
• performing, during runtime, a computation associated with the operator on the input data in the dense format to generate the output tensor, the output tensor storing output data along the plurality of dimensions in a dense format;
• and
• performing, during runtime, a reverse permutation to rearrange the output data in the output tensor from the dense format to a specified output data format;
• wherein the instructions to perform the permutation, the computation, and the reverse permutation are defined as primitives in a sparse kernel.

A non-transitory computer-readable medium storing a program executable by one or more processors, the program comprising sets of instructions for:

• identifying an operator in the NN model, the operator configured to perform a computation on at least one input tensor to generate an output tensor, the at least one input tensor storing input data along a plurality of dimensions in a sparse format;
• performing, during runtime, a permutation to rearrange the input data from the sparse format to a dense format, wherein the dense format is shorter than the sparse format along a dimension of the plurality of dimensions;
• performing, during runtime, a computation associated with the operator on the input data in the dense format to generate the output tensor, the output tensor storing output data along the plurality of dimensions in a dense format;
• and
• performing, during runtime, a reverse permutation to rearrange the output data in the output tensor from the dense format to a specified output data format;
• wherein the instructions to perform the permutation, the computation, and the reverse permutation are defined as primitives in a sparse kernel.