# US 12,602,252 B2 · Sparse matrix multiplication in a neural network > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,252 B2 - Document ID: US12602252B2 - Status: Granted - Kind: B2 - Application number: 18/240,281 - Filed: 2023-08-30 - Published: 2025-02-06 - Granted: 2026-04-14 - Earliest claimed: 2023-08-03 - Assignees: NVIDIA Corporation - Applicants: NVIDIA Corporation - Inventors: Jorge Albericio Latorre; Chong Yu - Web page: https://patentgrep.com/patent/US12602252B2 ## CPC - G06F 9/5027: {the resource being a machine, e.g. CPUs, Servers, Terminals} - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Apparatuses, systems, and methods to enable matrix multiplication acceleration by modifying an input to apply sparsity through sparse activation filtering. In at least one embodiment, a neural network modifies pixels within an image through sparse activation filtering to enable use of one or more matrix multiplication acceleration units to perform a sparse patch embedding operation. ## Claims ### Claim 1 A processor, comprising: one or more circuits to: modify one or more values corresponding to one or more pixels within one or more images to zero according to a sparsity constraint; and cause one or more operations to be performed, using one or more matrix multiplication acceleration units, on one or more operands representing pixels including the one or more modified pixels, wherein the one or more matrix multiplication acceleration units are configured to accelerate operation based on the sparsity constraint. ### Claim 2 (depends on 1) The processor of claim 1, wherein the one or more matrix multiplication acceleration units comprise a sparse tensor core, the sparse tensor core comprising circuitry to perform accelerated general matrix multiplication (GEMM). ### Claim 3 (depends on 1) The processor of claim 1, wherein the one or more matrix multiplication acceleration units are to prune and compress at least one matrix. ### Claim 4 (depends on 1) The processor of claim 1, wherein the one or more circuits are to determine an active portion and an inactive portion of an input image, and are to perform a patch embedding operation using the one or more matrix multiplication acceleration units based, at least in part, on the active portion of the input image. ### Claim 5 (depends on 4) The processor of claim 4, wherein the one or more circuits are to divide the input image into one or more square patches and convert the one or more square patches to a patch matrix comprising flattened row vectors representing the active portion and all-zero row vectors representing the inactive portion. ### Claim 6 (depends on 4) The processor of claim 4, wherein the one or more circuits are to determine the active portion and the inactive portion of the input image according to an activation threshold. ### Claim 7 (depends on 4) The processor of claim 4, wherein the patch embedding operation comprises using at least one index matrix to identify locations of non-zero values in a matrix used to generate vector tokens. ### Claim 8 (depends on 4) The processor of claim 4, wherein the patch embedding operation uses a row-sparse patch matrix and a sparse weight matrix to generate final patch embedding. ### Claim 9 A system, comprising: one or more processors to: modify one or more values corresponding to one or more pixels within one or more images to zero according to a sparsity constraint; and cause one or more operations to be performed, using one or more matrix multiplication acceleration units, on one or more operands representing pixels including the one or more modified pixels, wherein the one or more matrix multiplication acceleration units are configured to accelerate operation based on the sparsity constraint. ### Claim 10 (depends on 9) The system of claim 9, wherein the one or more matrix multiplication acceleration units comprises a sparse tensor core, the sparse tensor core comprising circuitry to perform accelerated general matrix multiplication (GEMM). ### Claim 11 (depends on 9) The system of claim 9, wherein the one or more matrix multiplication acceleration units prunes and compresses at least one matrix. ### Claim 12 (depends on 9) The system of claim 9, wherein the one or more processors are to determine an active portion and an inactive portion of an input image, and compute a patch embedding operation using the one or more matrix multiplication acceleration units based, at least in part, on the active portion of the input image. ### Claim 13 (depends on 12) The system of claim 12, wherein the input image is divisible into one or more square patches, wherein one or more portions corresponding to the one or more square patches are converted to a patch matrix comprising flattened row vectors representing the active portion and all-zero row vectors representing the inactive portion. ### Claim 14 (depends on 12) The system of claim 12, wherein the one or more processors are to use at least one index matrix to identify locations of non-zero values in a matrix used to generate vector tokens. ### Claim 15 A method, comprising: modifying one or more values corresponding to one or more pixels within one or more images to zero according to a sparsity constraint; and causing one or more operations to be performed, using one or more matrix multiplication acceleration units, on one or more operands representing pixels including the one or more modified pixels, wherein the one or more matrix multiplication acceleration units are configured to accelerate operation based on the sparsity constraint. ### Claim 16 (depends on 15) The method of claim 15, further comprising using the one or more matrix multiplication acceleration units to perform accelerated GEMM. ### Claim 17 (depends on 15) The method of claim 15, wherein the matrix multiplication acceleration unit prunes and compresses at least one matrix. ### Claim 18 (depends on 15) The method of claim 15, further comprising determining an active portion and an inactive portion of an input image, and compute a patch embedding operation using the one or more matrix multiplication acceleration units based, at least in part, on the active portion of the input image. ### Claim 19 (depends on 18) The method of claim 18, further comprising generating flattened row vectors representing the active portion and all-zero row vectors representing the inactive portion. ### Claim 20 (depends on 18) The method of claim 18, wherein the patch embedding operation includes using at least one index matrix to identify locations of non-zero values in a matrix used to generate vector tokens.