# US 12,762,394 B1 · Managing data sparsity for neural networks > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,762,394 B1 - Document ID: US12762394B1 - Status: Granted - Kind: B1 - Application number: 17/866,031 - Filed: 2022-07-15 - Granted: 2026-10-13 - Earliest claimed: 2018-05-02 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Jeff Pool; Ganesh Venkatesh; Jorge Albericio Latorre; Jack Choquette; Ronny Krashinsky; John Tran; Feng Xie; Ming Y. Siu; Manan Patel - Web page: https://patentgrep.com/patent/US12762394B1 ## CPC - G06N 3/08: Learning methods - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06N 3/04: Architecture, e.g. interconnection topology - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Approaches in accordance with various embodiments provide for the processing of sparse matrices for mathematical and programmatic operations. In particular, various embodiments enforce sparsity constraints for performing sparse matrix multiply-add instruction (MMA) operations. Deep neural networks can exhibit significant sparsity in the data used in operations, both in the activations and weights. The computational load can be reduced by excluding zero-valued data elements. A sparsity constraint is applied across all submatrices of a sparse matrix, providing fine-grained structured sparsity that is evenly distributed across the matrix. The matrix may then be compressed since a minimum number of elements of the matrix are known to have zero value. Matrix operations are then performed using these matrices. ## Claims ### Claim 1 A processor, comprising: one or more circuits to; identify a matrix multiplication operation subject to a sparsity constraint indicating a required minimum number of zero-values for a sparsity pattern repeated across one or more matrix operands of the matrix multiplication operation; and responsive to the one or more matrix operands satisfying the sparsity constraint, compress the one or more matrix operands of the matrix multiplication operation to exclude a plurality of zero-values from the one or more matrix operands according to the sparsity pattern. ### Claim 2 (depends on 1) The processor of claim 1, wherein the one or more matrix operands are operands in matrix multiply-add (MMA) operations, and wherein the one or more matrix operands correspond to at least one of weights or activations of one or more neural networks. ### Claim 3 (depends on 1) The processor of claim 1, wherein compressing the one or more matrix operands comprises reducing a number of rows or a number of columns of the one or more matrix operands. ### Claim 4 (depends on 3) The processor of claim 3, wherein the one or more circuits are further to store metadata corresponding to the plurality of zero values excluded from the one or more matrix operands. ### Claim 5 (depends on 1) The processor of claim 1, wherein the one or more circuits are further to: determine a plurality of submatrices of equal size and in accordance with sparsity pattern of the one or more matrix operands such that each element of a respective submatrix is contained in one and only one submatrix. ### Claim 6 (depends on 5) The processor of claim 5, wherein the plurality of submatrices are generated in a compressed sparse row (CSR) format, compressed sparse column (CSC), or a compressed sparse block (CSB) format. ### Claim 7 (depends on 1) The processor of claim 1, wherein the one or more circuits are further to ensure that a first operand of the one or more matrix operands corresponds to one or more sparse matrices and a second operand of the one or more matrix operands operations corresponds to one or more dense matrices. ### Claim 8 A system, comprising: one or more computers comprising one or more processors to: determine a matrix multiplication operation subject to a sparsity constraint indicating a required minimum number of zero-values for a sparsity pattern repeated across one or more matrix operands of the matrix multiplication operation; and responsive to the one or more matrix operands satisfying the sparsity constraint, compress the one or more matrix operands of the matrix multiplication operation to exclude a plurality of zero-values from the one or more matrix operands according to the sparsity pattern. ### Claim 9 (depends on 8) The system of claim 8, wherein the one or more matrix operands are operands in matrix multiply-add (MMA) operations, and wherein the one or more matrix operands correspond to at least one of weights or activations of one or more neural networks. ### Claim 10 (depends on 8) The system of claim 8, wherein the one or more processors are further to compress the one or more matrix operands by excluding one or more zero values from the one or more matrix operands and reducing a number of rows or a number of columns of the one or more matrix operands. ### Claim 11 (depends on 10) The system of claim 10, wherein the one or more processors are further to store metadata corresponding to the one or more zero values excluded from the one or more matrix operands. ### Claim 12 (depends on 8) The system of claim 8, wherein the one or more processors are further to determine a plurality of submatrices of the one or more matrix operands and further constrain the sparsity across the plurality of submatrices. ### Claim 13 (depends on 12) The system of claim 12, wherein the plurality of submatrices are generated in a compressed sparse row (CSR) format, or a compressed sparse column (CSC) format, or a compressed sparse block (CSB). ### Claim 14 (depends on 8) The system of claim 8, wherein the one or more processors are further to ensure that a first operand of the one or more matrix multiply operands corresponds to one or more sparse matrices and a second operand of the one or more matrix multiply operands corresponds to one or more dense matrices. ### Claim 15 A method, comprising: identifying a matrix multiplication operation subject to a sparsity constraint indicating a required minimum number of zero-values for a sparsity pattern repeated across one or more matrix operands of the matrix multiplication operation; and responsive to the one or more matrix operands satisfying the sparsity constraint, compressing the one or more matrix operands of the matrix multiplication operation to exclude a plurality of zero-values from the one or more matrix operands according to the sparsity pattern. ### Claim 16 (depends on 15) The method of claim 15, wherein the one or more matrix operands are operands in matrix multiply-add (MMA) operations, and wherein the one or more matrix operands correspond to at least one of weights or activations of one or more neural networks. ### Claim 17 (depends on 15) The method of claim 15, wherein compressing the one or more matrix operands comprises reducing a number of rows or a number of columns of the one or more matrix operands. ### Claim 18 (depends on 17) The method of claim 17, further comprising storing metadata corresponding to the one or more zero values excluded from the one or more matrix operands. ### Claim 19 (depends on 15) The method of claim 15, further comprising determining a plurality of submatrices of the one or more matrix operands, and further constraining the sparsity across the plurality of submatrices. ### Claim 20 (depends on 15) The method of claim 15, further comprising ensuring that a first operand of the one or more matrix operands corresponds to one or more sparse matrices and a second operand of the one or more matrix operands corresponds to one or more dense matrices.