# US 2023/0267171 A1 · Low latency matrix multiply unit

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0267171 A1
- Document ID: US20230267171A1
- Status: Published application
- Kind: A1
- Application number: 17/985,069
- Filed: 2022-11-10
- Published: 2023-08-24
- Earliest claimed: 2017-05-17
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Andrew Everett Phelps; Norman Paul Jouppi
- Web page: https://patentgrep.com/patent/US20230267171A1

## CPC

- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 5/015: {having at least two separately controlled shifting levels, e.g. using shifting matrices (G06F5/012 takes precedence)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/30101: {Special purpose registers}
- G06F 15/8046: {Systolic arrays}
- G06N 3/063: using electronic means
- G06F 9/30032: {Movement instructions, e.g. MOVE, SHIFT, ROTATE, SHUFFLE}
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/08: Learning methods
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. The matrix multiply unit may include cells arranged in columns of the systolic array. Two chains of weight shift registers per column of the systolic array are in the matrix multiply unit. Each weight shift register is connected to only one chain and each cell is connected to only one weight shift register. A weight matrix register per cell is configured to store a weight input received from a weight shift register. A multiply unit is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result.

## Claims

### Claim 1

(canceled)

### Claim 2

A matrix multiply unit implemented as a two-dimensional systolic array comprising:
a plurality of cells arranged in columns of the systolic array;
at least one chain of weight shift registers per column of the systolic array; wherein each weight shift register is connected to only one chain and each cell is connected to only one weight shift register of the at least one chain, the at least one chain having two injection points for injecting weight values, one at the top of the column, and the other at a second point in the column;
a weight matrix register per cell configured to store a weight input received from a weight shift register; and
a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input to obtain a multiplication result.

### Claim 3 (depends on 2)

The matrix multiply unit of claim 2, wherein the other one of the two injection points is at a halfway point of the column or at a point one-fourth of the way down the column.

### Claim 4 (depends on 2)

The matrix multiply unit of claim 2, further comprising a multiplexer on the at least one chain at the second point in the column configured to select a weight value from the other injection point or a weight shift register of the at least one chain.

### Claim 5 (depends on 2)

The matrix multiply unit of claim 2, comprising two chains of weight shift registers per column of the systolic array; wherein each weight shift register is connected to only one chain, each chain having two injection points for injecting weight values.

### Claim 6 (depends on 5)

The matrix multiply unit of claim 5, further comprising:
a vector register containing packed sets of four 8-bit integers each representing a separate weight value.

### Claim 7 (depends on 6)

The matrix multiply unit of claim 6, further comprising:
injecting two of the four 8-bit integers at the top of the column and injecting the other two of the four 8-bit integers to the second point in the array.

### Claim 8 (depends on 6)

The matrix multiply unit of claim 6, further comprising a holding register at the top of each column to hold a weight value when two weight values are unavailable from the vector register.

### Claim 9 (depends on 8)

The matrix multiply unit of claim 8, wherein when two weight values are unavailable:
on a first clock cycle that a first weight value is available, the holding register is loaded with the first weight value as a held value and no shifting is done; and
on a next clock cycle, when a second weight value is available, the second weight value and the held value are shifted, by the two weight shift register chains, one value shifted by each weight shift register chain, to weight shift registers connected to the two weight shift register chains.

### Claim 10 (depends on 8)

The matrix multiply unit of claim 8, wherein when two weight values are available, the two weight values are shifted on a clock cycle to the weight shift registers in the cells.

### Claim 11 (depends on 2)

The matrix multiply unit of claim 2, wherein when data is in the weight matrix register, the data is used in any number of cycles of multiplications.

### Claim 12 (depends on 11)

The matrix multiply unit of claim 11, wherein during the any number of cycles of multiplications, more weights are shifted into the weight shift registers in the background in preparation for the next set of multiplications.

### Claim 13 (depends on 11)

The matrix multiply unit of claim 11, wherein during the any number of cycles of multiplications, a weight input of the weight matrix register is multiplied with a vector data input to obtain a multiplication result.

### Claim 14 (depends on 2)

The matrix multiply unit of claim 2, comprising a plurality of rows of the systolic array, each row comprising a plurality of cells, further comprising:
at least one further chain of further weight shift registers per row of the systolic array; wherein each further weight shift register is connected to only one further chain of the at least one further chain and each cell is connected to only one further weight shift register, the at least one further chain having two injection points for injecting weight values.

### Claim 15

A matrix multiply unit implemented as a two-dimensional systolic array comprising:
a plurality of rows of the systolic array, each row comprising a plurality of cells;
at least one chain of weight shift registers per row of the systolic array; wherein each weight shift register is connected to only one chain and each cell is connected to only one weight shift register, the at least one chain having two injection points for injecting weight values, one at a first end of the row, and the other at a second point in the row;
a weight matrix register per cell configured to store a weight input received from a weight shift register; and
a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input to obtain a multiplication result.

### Claim 16 (depends on 15)

The matrix multiply unit of claim 15, comprising two chains of weight shift registers per row of the systolic array; wherein each weight shift register of the two chains of weight shift registers per row is connected to only one chain, each chain having two injection points for injecting weight values.

### Claim 17 (depends on 15)

The matrix multiply unit of claim 15, wherein the other one of the two injection points is at a halfway point of the row or at a point one-fourth the way from the first end of the row.

### Claim 18 (depends on 15)

The matrix multiply unit of claim 15, further comprising a multiplexer on the at least one chain at the second point in the row configured to select a weight value from the other injection point or a weight shift register of the at least one chain.

### Claim 19 (depends on 15)

The matrix multiply unit of claim 15, wherein when data is in the weight matrix register, the data is used in any number of cycles of multiplications.

### Claim 20 (depends on 19)

The matrix multiply unit of claim 19, wherein during the any number of cycles of multiplications, more weights are shifted into the weight shift registers in the background in preparation for the next set of multiplications.

### Claim 21 (depends on 19)

The matrix multiply unit of claim 19, wherein during the any number of cycles of multiplications, a weight input of the weight matrix register is multiplied with a vector data input to obtain a multiplication result.
