# US 11,907,330 B2 · Low latency matrix multiply unit > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,907,330 B2 - Document ID: US11907330B2 - Status: Granted - Kind: B2 - Application number: 18/111,468 - Filed: 2023-02-17 - Published: 2023-08-24 - Granted: 2024-02-20 - Earliest claimed: 2017-05-17 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Andrew Everett Phelps; Norman Paul Jouppi - Web page: https://patentgrep.com/patent/US11907330B2 ## CPC - G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)} - G06F 5/015: {having at least two separately controlled shifting levels, e.g. using shifting matrices (G06F5/012 takes precedence)} - G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)} - G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations} - G06F 9/30101: {Special purpose registers} - G06F 15/8046: {Systolic arrays} - G06N 3/063: using electronic means - G06F 9/30032: {Movement instructions, e.g. MOVE, SHIFT, ROTATE, SHUFFLE} - G06N 3/04: Architecture, e.g. interconnection topology - G06N 3/08: Learning methods - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract Methods, systems, and apparatus for a matrix multiply unit implemented as a systolic array of cells are disclosed. Each cell of the matrix multiply includes: a weight matrix register configured to receive a weight input from either a transposed or a non-transposed weight shift register; a transposed weight shift register configured to receive a weight input from a horizontal direction to be stored in the weight matrix register; a non-transposed weight shift register configured to receive a weight input from a vertical direction to be stored in the weight matrix register; and a multiply unit that is coupled to the weight matrix register and configured to multiply the weight input of the weight matrix register with a vector data input in order to obtain a multiplication result. ## Claims ### Claim 1 A processor comprising: one or more cores, each core having an array of cells, and one or more storage devices on which are stored instructions that are operable, when executed by a core of the one or more cores, to cause the core to perform operations comprising: receiving data representing instructions for performing a portion of operations; generating instructions that, for each cell of an array of cells included in the core and once executed by the cell, causes the cell to perform operations comprising: receiving, by a weight matrix register of the cell, a weight input from a plurality of weight storing registers, wherein each of the plurality of weight storing registers is configured to receive a respective weight input from a respective direction of the array, wherein the respective directions comprise a first direction of the array and a second direction of the array different from the first direction; and obtaining, by a multiply unit that is coupled to the weight matrix register, a respective multiplication result based on the received weight input and another input; and generating a result for performing the portion of operations by combining the respective multiplication results obtained from the array of cells. ### Claim 2 (depends on 1) The processor of claim 1, wherein the portion of operations is associated with a network layer of a neural network, the respective weight input is associated with the network layer, and the other input includes an activation input of the network layer. ### Claim 3 (depends on 2) The processor of claim 2, wherein the operations performed by the cell further comprises: receiving, by the multiply unit, another weight input from the weight matrix register; and multiplying, by the multiply unit, the other received weight input with another activation input associated with the network layer to generate another multiplication result. ### Claim 4 (depends on 1) The processor of claim 1, wherein the array of cells is a systolic array of cells, and the array has a two-dimensional format, wherein the first direction of the array is the first direction in the two-dimensional format, and the second direction of the array is the second direction in the two-dimensional format. ### Claim 5 (depends on 1) The processor of claim 1, wherein receiving, by the weight matrix register of the cell, the weight input from the plurality of weight storing registers further comprises: selecting, by a multiplexer of the cell, the weight input from at least two respective weight inputs each stored in a corresponding weight storing register of the plurality of weight storing registers; and receiving, by the weight matrix register, the selected weight input from the multiplexer. ### Claim 6 (depends on 1) The processor of claim 1, wherein the plurality of weight storing registers comprise a transposed weight shift register and a non-transposed weight shift register physically separate from the transposed weight shift register. ### Claim 7 (depends on 1) The processor of claim 1, wherein the plurality of weight storing registers comprise: a first weight storing register configured to receive a first weight input over a first wired path from a first cell of the array of cells that is along the first direction; and a second weight storing register configured to receive a second weight input over a second wired path from a second cell of the array of cells that is along the second direction; wherein the weight input is one of the first weight input and the second weight input. ### Claim 8 (depends on 1) The processor of claim 1, wherein the core further includes one or more of a scalar memory, a vector memory, a scalar processing unit, one or more vector registers, a transposed unit, or a reduction and permutation unit. ### Claim 9 A method performed by a core of one or more cores included in a processor, the method comprising: receiving data representing instructions for performing a portion of operations; generating instructions that, for each cell of an array of cells included in the core and once executed by the cell, causes the cell to perform operations comprising: receiving, by a weight matrix register of the cell, a weight input from a plurality of weight storing registers, wherein each of the plurality of weight storing registers is configured to receive a respective weight input from a respective direction of the array, wherein the respective directions comprise a first direction of the array and a second direction of the array different from the first direction; and obtaining, by a multiply unit that is coupled to the weight matrix register, a respective multiplication result based on the received weight input and another input; and generating a result for performing the portion of operations by combining the respective multiplication results obtained from the array of cells. ### Claim 10 (depends on 9) The method of claim 9, wherein the portion of operations is associated with a network layer of a neural network, the respective weight input is associated with the network layer, and the other input includes an activation input of the network layer. ### Claim 11 (depends on 10) The method of claim 10, wherein the operations performed by the cell further comprises: receiving, by the multiply unit, another weight input from the weight matrix register; and multiplying, by the multiply unit, the other received weight input with another activation input associated with the network layer to generate another multiplication result. ### Claim 12 (depends on 10) The method of claim 10, wherein the array of cells is a systolic array of cells, and the array has a two-dimensional format, wherein the first direction of the array is the first direction in the two-dimensional format, and the second direction of the array is the second direction in the two-dimensional format. ### Claim 13 (depends on 10) The method of claim 10, wherein receiving, by the weight matrix register of the cell, the weight input from the plurality of weight storing registers further comprises: selecting, by a multiplexer of the cell, the weight input from at least two respective weight inputs each stored in a corresponding weight storing register of the plurality of weight storing registers; and receiving, by the weight matrix register, the selected weight input from the multiplexer. ### Claim 14 (depends on 10) The method of claim 10, wherein the plurality of weight storing registers comprise a transposed weight shift register and a non-transposed weight shift register physically separate from the transposed weight shift register. ### Claim 15 (depends on 10) The method of claim 10, wherein the plurality of weight storing registers comprise: a first weight storing register configured to receive a first weight input over a first wired path from a first cell of the array of cells that is along the first direction; and a second weight storing register configured to receive a second weight input over a second wired path from a second cell of the array of cells that is along the second direction; wherein the weight input is one of the first weight input and the second weight input. ### Claim 16 (depends on 10) The method of claim 10, wherein the core further includes one or more of a scalar memory, a vector memory, a scalar processing unit, one or more vector registers, a transposed unit, or a reduction and permutation unit. ### Claim 17 A non-transitory computer program product storing instructions that, when executed by a core of one or more cored included in a programmable processor, cause the core to perform operations comprising: receiving data representing instructions for performing a portion of operations; generating instructions that, for each cell of an array of cells included in the core and once executed by the cell, causes the cell to perform operations comprising: receiving, by a weight matrix register of the cell, a weight input from a plurality of weight storing registers, wherein each of the plurality of weight storing registers is configured to receive a respective weight input from a respective direction of the array, wherein the respective directions comprise a first direction of the array and a second direction of the array different from the first direction; and obtaining, by a multiply unit that is coupled to the weight matrix register, a respective multiplication result based on the received weight input and another input; and generating a result for performing the portion of operations by combining the respective multiplication results obtained from the array of cells. ### Claim 18 (depends on 17) The non-transitory computer program product of claim 17, wherein the portion of operations is associated with a network layer of a neural network, the respective weight input is associated with the network layer, and the other input includes an activation input of the network layer. ### Claim 19 (depends on 18) The non-transitory computer program product of claim 18, wherein the operations performed by the cell further comprises: receiving, by the multiply unit, another weight input from the weight matrix register; and multiplying, by the multiply unit, the other received weight input with another activation input associated with the network layer to generate another multiplication result. ### Claim 20 (depends on 17) The non-transitory computer program product of claim 17, wherein receiving, by the weight matrix register of the cell, the weight input from the plurality of weight storing registers further comprises: selecting, by a multiplexer of the cell, the weight input from at least two respective weight inputs each stored in a corresponding weight storing register of the plurality of weight storing registers; and receiving, by the weight matrix register, the selected weight input from the multiplexer.