# US 12,762,203 B2 · System and method for implementing variable-precision matrix multiplication using low-precision digit matrix multiplier

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,762,203 B2
- Document ID: US12762203B2
- Status: Granted
- Kind: B2
- Application number: 17/737,073
- Filed: 2022-05-05
- Published: 2023-11-09
- Granted: 2026-10-13
- Earliest claimed: 2022-05-05
- Assignees: Xcelerium Inc
- Applicants: Xcelerium Inc
- Inventors: Hamza Khan; Asma Khan; Raheel Khan
- Web page: https://patentgrep.com/patent/US12762203B2

## CPC

- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 7/523: Multiplying only
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A system and method for implementing variable-precision matrix multiplications using a low-precision digit matrix multiplier is disclosed. The system enables multiplication of matrices of different dimensions by splitting the large matrix into fixed-size matrix blocks. These block matrices are further decomposed into fixed-precision digit submatrices that are then individually multiplied, scaled, and accumulated to allow for variable-precision matrix multiplication. The system uses a systolic array of block matrix multipliers, which are each an array of dot product units, to efficiently implement larger matrix multiplications without substantially increasing either latency or wiring congestion. The system further uses only unsigned digit matrix multipliers but accounts for signed matrix multiplication by using row and column sums of the input matrices to adjust for the signed to unsigned conversion.

## Claims

### Claim 1

A system for implementing variable-precision matrix operations that uses an array of low-precision (1, 2, and 4 bits) unsigned integer dot product units and an array of higher-precision accumulators to calculate a matrix product of higher-precision integer elements, the system comprising:
one or more hardware processors;
a computer readable medium coupled to the one or more hardware processors, wherein the computer readable medium comprises a set of program instructions in a form of a plurality of subsystems, configured to be executed by the one or more hardware processors, wherein the plurality of subsystems comprises:
a load store unit configured to load and store data between a register file and a system memory, the system memory being one of: external and on-chip;
the register file configured to provide the data in natural order to a permute unit, wherein the matrices are stored in a natural order with each element in a row stored contiguously;
the permute unit comprising circuitry configured to deinterleave the data, wherein the deinterleaving comprises converting a matrix of multi-digit elements into multiple matrices with single-digit elements of same weight stored contiguously, and converting the data into fixed dimension and fixed precision sub-matrices, wherein the deinterleaved data is stored back into the register file; and
a Digit Matrix Multiplier Unit configured to receive digit matrices in deinterleaved form from the register file, wherein the Digit Matrix Multiplier Unit comprises a plurality of subsystems to perform the following operations:
an input unit comprising one or more buffers and circuitry configured to decode instructions and generate control signals, wherein concurrently, the digit matrices are buffered, and a portion of the digit matrices are transmitted to a compute unit to be multiplied;
the compute unit comprising a systolic array of processing elements, each processing element comprising an array of dot product units, wherein each dot product unit configured to compute a product of two digit matrices and to transmit the product to a matrix combination unit to be accumulated wherein each dot product unit computes a dot-product of one row and one column of the digit matrix;
the matrix combination unit comprising an accumulator array and a shifter, configured to accumulate results of current digit matrix multiplications with results of previous multiplications and apply scale factors to account for multi-precision computations, and apply offsets to account unsigned product to signed product conversion, wherein the accumulation results are stored locally inside the matrix combination unit and transmits a portion of the results to an output unit; and
the output unit comprising circuitry configured to convert full-precision elements of the matrix product to a reduced precision by rounding off the full precision matrix multiply results, shifting the rounded results, and saturating the shifted results to a desired precision.

### Claim 2 (depends on 1)

The system of claim 1, wherein the Digit Matrix Multiplier Unit uses unsigned digit matrix multiplication to calculate a product of two signed matrices and adjusts for sign correction using a sum of all elements within each row of one matrix and a sum of all elements within each column of the other matrix, wherein the row sums and column sums are calculated using dedicated hardware blocks.

### Claim 3

A computer-implemented method for implementing variable-precision matrix operations using a low-precision (1-, 2-, and 4-bit) digit matrix multiplication, the method comprising:
(a) decomposing by a permute unit, a first matrix A into a set {Air} of n×n sub-matrices of M-bit elements, wherein the first matrix A is an N1×N matrix, and wherein i={0, . . . , N1/n−1} and r={0, . . . , N/n−1};
(b) decomposing by the permute unit, a second matrix B into a set {Brj} of n×n sub-matrices of M-bit elements, wherein the second matrix B is an N×N2 matrix, and wherein r={0, . . . , N/n−1} and j={0, . . . , N2/n−1};
(c) initializing i and j to 0;
(d) computing product sub-matrix Cij, wherein the computation comprises:
(d1) initializing r to 0 and initializing, by a matrix combination unit, n×n bank of accumulators ACC to 0;
(d2) retrieving, by a load-store unit, the submatrix, Air and the submatrix Brj from one of a subset comprising on-chip and off-chip shared memory, and then placing the submatrix into an intermediate storage location; and
(d3) computing product Air×Brj, wherein the computing comprises:
(d3i) decomposing, by the permute unit, Air into a set of K m-bit digit matrices
{
A
ir
k
⁢
0
}
,
wherein K=M/m and digit position k0 ranges from {0, . . . , K−1}, and wherein
A
ir
k
⁢
0
has a corresponding scale factor 2m*k0, and wherein the digit matrix
{
A
ir
k
⁢
0
}
is placed in the intermediate storage location;
(d3ii) decomposing, by the permute unit, Brj into a set of K m-bit digit matrices
{
B
rj
k
⁢
1
}
,
wherein K=M/m and digit position k1={0, . . . , K−1}, and wherein
B
rj
k
⁢
1
has the corresponding scale factor 2m*k1, and wherein the digit matrix
{
B
rj
k
⁢
1
}
is placed in the intermediate storage location;
(d3iii) initializing k0 and k1 to 0 and setting a scale factor S to 1;
(d3iv) retrieving
A
ir
k
⁢
0
⁢
and
⁢
B
rj
k
⁢
1
from the intermediate storage location;
(d3v) computing, by a compute unit comprising a systolic array of processing elements, a third n×n matrix T by multiplying
A
ir
k
⁢
0
⁢
and
⁢
B
rj
k
⁢
1
using digit matrix multiplication;
(d3vi) multiplying, by a digit matrix multiplier functional unit, the third matrix T by the scale factor S and adding a respective product to the accumulator (ACC), wherein the accumulator is updated for resulting value ACC′=ACC+T*S;
(d3vii) incrementing, by the digit matrix multiplier functional unit, k1 and setting the scale factor S equal to 2m*(k0+k1) and going back to step d3iv provided that k1 is less than K otherwise going to step d3viii;
(d3viii) setting, by the digit matrix multiplier functional unit, k1 to 0, incrementing k0 and setting the scale factor S equal to 2m*(k0+k1) and going back to step d3iv provided that k0 is less than K otherwise going to step d3ix;
(d3ix) incrementing, by the digit matrix multiplier functional unit, the value of r and going back to step d2 provided that r is less than N/n otherwise going to step e;
(e) post processing, by an output unit, each accumulator element by:
applying a defined scale factor to each accumulator element comprising Cij;
performing rounding and saturating of a scaled element of Cij; and
storing the resulting scaled element in either the on-chip or off-chip shared memory;
(f) incrementing j and going back to step d provided that j is less than N2/n, otherwise going to step g; and
(g) setting j to 0, incrementing i, and going back to step d provided that i is less than N1/n.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein matrix A and matrix B are broken into rectangular sub-matrices of dimensions n1×n2 and n2×n3 respectively.

### Claim 5 (depends on 3)

The computer-implemented method of claim 3, wherein matrix A and matrix B have different element sizes M1 and M2 respectively.

### Claim 6 (depends on 3)

The computer-implemented method of claim 3, wherein the order of loops in step d3 (incrementing k0 and k1) is changed when computing the product of
A
i
⁢
r
k
⁢
0
and
B
r
⁢
j
k
⁢
1
.

### Claim 7 (depends on 3)

The computer-implemented method of claim 3, wherein the order of the loop in step d (incrementing r) is changed when computing the product Cij.

### Claim 8 (depends on 3)

The computer-implemented method of claim 3, wherein the order of loops (incrementing i and j) is changed when computing the product of Air and Brj.