Weight-sparse neural processing unit with multi-dimensional routing of non-zero values

A general matrix-matrix (GEMM) accelerator core, comprising:

• a first buffer comprising K0 rows and K1 columns of locations, the first buffer being configured to receive a elements of a first matrix A of activation values, and K0 and K1 being integers greater than 1;
• a second buffer comprising K1 rows and K0 columns of locations, the second buffer being configured to receive b elements of a second matrix B of weight values, the second matrix B being preprocessed with a first nonzero-valued b element replacing a first zero-valued b element in a first row of the second buffer based on the first zero-valued b element being in the first row of the second buffer and the first nonzero-valued b element being within a maximum borrowing distance of a first location of the first zero-valued b element in the first row of the second buffer, the preprocessing further generating metadata that includes movement information of the first nonzero-valued b element to replace the first zero-valued b element;
• and
• a first processing element (PE) comprising an array of K0 multipliers, the first PE being associated with the first buffer and the second buffer, the first PE being configured to receive b elements from a first row of the second buffer and a elements from the first buffer from locations in the first buffer that correspond to locations in the second buffer from where the b elements have been received by the first PE as indicated by the metadata.

A general matrix-matrix (GEMM) accelerator core, comprising:

• a first buffer comprising K0 rows and K1 columns of locations, the first buffer being configured to receive a elements of a first matrix A of activation values, and K0 and K1 being integers greater than 1;
• a second buffer comprising K1 rows and K0 columns of locations, the second buffer being configured to receive b elements of a second matrix B of weight values, the second matrix B being preprocessed with a first nonzero-valued b element replacing a first zero-valued b element in a first row of the second buffer based on the first zero-valued b element being in the first row of the second buffer, the preprocessing further generating metadata that includes movement information of the first nonzero-valued b element to replace the first zero-valued b element;
• a third buffer comprising K1 rows and K0 columns of locations, the third buffer configured to receive b elements of the second matrix B of weight values, the second matrix B being further preprocessed with a second nonzero-valued b element replacing a second zero-valued b element in a first row of the third buffer based on the second zero-valued b element being in the first row of the third buffer, the preprocessing further generating metadata that includes movement information of the second nonzero-valued b element to replace the second zero-valued b element;
• a first processing element (PE) comprising an array of K0 multipliers, the first PE being associated with the first buffer and the second buffer, the first PE being configured to receive b elements from a first row of the second buffer and a elements from the first buffer from locations in the first buffer that correspond to locations in the second buffer from where the b elements have been received by the first PE as indicated by the metadata;
• and
• a second PE comprising an array of K0 multipliers, the second PE being associated with the second buffer and the third buffer, the second PE being configured to receive b elements from the first row of the third buffer including the second nonzero-valued b element selected to replace the second zero-value b element and to receive a elements from locations in the first buffer that correspond to locations in the second and the third buffers from where the b elements have been received by the second PE.