# US 12,118,057 B2 · Computing partial matrices at hardware accelerator

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,118,057 B2
- Document ID: US12118057B2
- Status: Granted
- Kind: B2
- Application number: 17/149,619
- Filed: 2021-01-14
- Published: 2022-08-18
- Granted: 2024-10-15
- Earliest claimed: 2021-01-14
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Derek Edward Davout Gladding; Nitin Naresh Garegrat; Timothy Hume Heil; Balamurugan Kulanthivelu Veluchamy
- Web page: https://patentgrep.com/patent/US12118057B2

## CPC

- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06N 3/063: using electronic means
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A computing device, including a hardware accelerator configured to receive a first matrix and receive a second matrix. The hardware accelerator may, for a plurality of partial matrix regions, in a first iteration, read a first submatrix of the first matrix and a second submatrix of the second matrix into a front-end processing area. The hardware accelerator may multiply the first submatrix by the second submatrix to compute a first intermediate partial matrix. In each of one or more subsequent iterations, the hardware accelerator may read an additional submatrix into the front end processing area. The hardware accelerator may compute an additional intermediate partial matrix as a product of the additional submatrix and a submatrix reused from an immediately prior iteration. The hardware accelerator may compute each partial matrix as a sum of two or more of the intermediate partial matrices and may output the plurality of partial matrices.

## Claims

### Claim 1

A computing device comprising:
a hardware accelerator including an on-chip controller configured to control the hardware accelerator to:
receive a first matrix at a first input buffer;
receive a second matrix at a second input buffer;
for a plurality of partial matrix regions each including a plurality of partial matrices:
in a first iteration of a plurality of iterations:
read, into a front-end processing area included in the hardware accelerator and operatively coupled to the first input buffer and the second input buffer, a first submatrix of the first matrix and a second submatrix of the second matrix; and
at the front-end processing area, multiply the first submatrix by the second submatrix to compute a first intermediate partial matrix of a plurality of intermediate partial matrices;
in each of one or more subsequent iterations of the plurality of iterations:
read an additional submatrix into the front-end processing area; and
at the front-end processing area, compute an additional intermediate partial matrix of the plurality of intermediate partial matrices as a product of the additional submatrix and a submatrix reused from an immediately prior iteration of the plurality of iterations, wherein the hardware accelerator is configured to alternate between reusing respective submatrices of the first matrix and the second matrix;
at a back-end processing area included in the hardware accelerator and operatively coupled to the front-end processing area, compute each partial matrix of the plurality of partial matrices as a sum of two or more of the intermediate partial matrices that correspond to a shared position within the partial matrix region; and
output the plurality of partial matrices to one or more other components of the computing device via a result buffer.

### Claim 2 (depends on 1)

The computing device of claim 1, wherein each partial matrix is a square matrix.

### Claim 3 (depends on 2)

The computing device of claim 2, wherein:
the first submatrix is a k×r matrix, where k and r are integers;
the second submatrix is an r×k matrix; and
each partial matrix is a k×k matrix.

### Claim 4 (depends on 1)

The computing device of claim 1, wherein each partial matrix region includes a rectangular grid of partial matrices.

### Claim 5 (depends on 4)

The computing device of claim 4, wherein, for each partial matrix region of the plurality of partial matrix regions, the hardware accelerator is configured to compute the plurality of intermediate partial matrices in a clockwise or counterclockwise cycle.

### Claim 6 (depends on 1)

The computing device of claim 1, wherein the plurality of partial matrix regions are each of a same size.

### Claim 7 (depends on 1)

The computing device of claim 1, wherein the hardware accelerator is further configured to round one or more partial matrix elements included in the plurality of partial matrices.

### Claim 8 (depends on 1)

The computing device of claim 1, wherein the hardware accelerator is configured to, at the back-end processing area, add a scalar to each partial matrix element included in the plurality of partial matrices.

### Claim 9 (depends on 1)

The computing device of claim 1, wherein the hardware accelerator is configured to, at the back-end processing area:
set at least one partial matrix element to a lower-bound value when the at least one partial matrix element is below a first threshold value; or
set the at least one partial matrix element to an upper-bound value when the at least one partial matrix element is above a second threshold value.

### Claim 10 (depends on 1)

The computing device of claim 1, wherein a plurality of first matrices and a plurality of second matrices in which the first matrix and the second matrix are respectively included are multiplied in parallel at a respective plurality of hardware accelerators included in a hardware accelerator array.

### Claim 11

A method for use with a computing device, the method comprising:
at an on-chip controller included in a hardware accelerator, controlling the hardware accelerator to:
receiving a first matrix at a first input buffer;
receiving a second matrix at a second input buffer;
for a plurality of partial matrix regions each including a plurality of partial matrices:
in a first iteration of a plurality of iterations:
reading, into a front-end processing area included in the hardware accelerator and operatively coupled to the first input buffer and the second input buffer, a first submatrix of the first matrix and a second submatrix of the second matrix; and
at the front-end processing area, multiplying the first submatrix by the second submatrix to compute a first intermediate partial matrix of a plurality of intermediate partial matrices;
in each of one or more subsequent iterations of the plurality of iterations:
reading an additional submatrix into the front-end processing area; and
at the front-end processing area, computing an additional intermediate partial matrix of the plurality of intermediate partial matrices as a product of the additional submatrix and a submatrix reused from an immediately prior iteration of the plurality of iterations, wherein the hardware accelerator alternates between reusing respective submatrices of the first matrix and the second matrix;
at a back-end processing area included in the hardware accelerator and operatively coupled to the front-end processing area, computing each partial matrix of the plurality of partial matrices as a sum of two or more of the intermediate partial matrices that correspond to a shared position within the partial matrix region; and
outputting the plurality of partial matrices to one or more other components of the computing device via a result buffer.

### Claim 12 (depends on 11)

The method of claim 11, wherein each partial matrix is a square matrix.

### Claim 13 (depends on 12)

The method of claim 12, wherein:
the first submatrix is a k×r matrix, where k and r are integers;
the second submatrix is an r×k matrix; and
each partial matrix is a k×k matrix.

### Claim 14 (depends on 11)

The method of claim 11, wherein each partial matrix region includes a rectangular grid of partial matrices.

### Claim 15 (depends on 11)

The method of claim 11, wherein the plurality of partial matrix regions are each of a same size.

### Claim 16 (depends on 11)

The method of claim 11, further comprising, at the hardware accelerator, rounding one or more partial matrix elements included in the plurality of partial matrices.

### Claim 17 (depends on 11)

The method of claim 11, further comprising, at the back-end processing area, adding a scalar to each partial matrix element included in the plurality of partial matrices.

### Claim 18 (depends on 11)

The method of claim 11, further comprising, at the back-end processing area:
setting at least one partial matrix element to a lower-bound value when the at least one partial matrix element is below a first threshold value; or
setting the at least one partial matrix element to an upper-bound value when the at least one partial matrix element is above a second threshold value.

### Claim 19 (depends on 11)

The method of claim 11, wherein a plurality of first matrices and a plurality of second matrices in which the first matrix and the second matrix are respectively included are multiplied in parallel at a respective plurality of hardware accelerators included in a hardware accelerator array.

### Claim 20

A computing device comprising:
a hardware accelerator including an on-chip controller configured to control the hardware accelerator to:
receive a first matrix at a first input buffer;
receive a second matrix at a second input buffer;
for a plurality of partial matrix regions each including a plurality of partial matrices, compute the plurality of partial matrices at least in part by:
in a first iteration of a plurality of iterations, at a front-end processing area included in the hardware accelerator and operatively coupled to the first input buffer and the second input buffer, multiplying a first submatrix including a plurality of first matrix elements of the first matrix by a second submatrix including a plurality of second matrix elements of the second matrix to compute a first intermediate partial matrix of a plurality of intermediate partial matrices;
in each of one or more subsequent iterations of the plurality of iterations, computing an additional intermediate partial matrix of the plurality of intermediate partial matrices as a product of an additional submatrix and a submatrix reused from an immediately prior iteration of the plurality of iterations, wherein the hardware accelerator is configured to alternate between reusing respective submatrices of the first matrix and the second matrix;
at a back-end processing area included in the hardware accelerator and operatively coupled to the front-end processing area, computing each partial matrix of the plurality of partial matrices as a sum of two or more of the intermediate partial matrices that correspond to a shared position within the partial matrix region; and
outputting an output matrix including the plurality of partial matrices to one or more other components of the computing device via a result buffer.