# US 2018/0321938 A1 · Generalized acceleration of matrix multiply accumulate operations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0321938 A1
- Document ID: US20180321938A1
- Status: Published application
- Kind: A1
- Application number: 15/826,435
- Filed: 2017-11-29
- Published: 2018-11-08
- Earliest claimed: 2017-05-08
- Assignees: NVIDIA Corporation
- Applicants: NVIDIA Corporation
- Inventors: Brent Ralph Boswell; Ming Y. Siu; Jack H. Choquette; Jonah M. Alben; Stuart Oberman
- Web page: https://patentgrep.com/patent/US20180321938A1

## CPC

- G06F 9/30014: {with variable precision}
- G06F 9/3001: {Arithmetic instructions}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3012: {Organisation of register space, e.g. banked or distributed register file}
- G06F 9/3851: {from multiple instruction streams, e.g. multistreaming}
- G06F 9/3888: {controlled by a single instruction for multiple threads [SIMT] in parallel}
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

A method, computer readable medium, and processor are disclosed for performing matrix multiply and accumulate (MMA) operations. The processor includes a datapath configured to execute the MMA operation to generate a plurality of elements of a result matrix at an output of the datapath. Each element of the result matrix is generated by calculating at least one dot product of corresponding pairs of vectors associated with matrix operands specified in an instruction for the MMA operation. A dot product operation includes the steps of: generating a plurality of partial products by multiplying each element of a first vector with a corresponding element of a second vector; aligning the plurality of partial products based on the exponents associated with each element of the first vector and each element of the second vector; and accumulating the plurality of aligned partial products into a result queue utilizing at least one adder.

## Claims

### Claim 1

A processor, comprising:
a datapath configured to execute a matrix multiply and accumulate (MMA) operation to generate a plurality of elements of a result matrix at an output of the datapath,
wherein each element in the plurality of elements of the result matrix is generated by calculating at least one dot product of corresponding pairs of vectors associated with matrix operands specified in an instruction for the MMA operation, and
wherein a dot product operation for calculating each dot product in the at least one dot product comprises:
generating a plurality of partial products by multiplying each element of a first vector with a corresponding element of a second vector,
aligning the plurality of partial products based on the exponents associated with each element of the first vector and each element of the second vector, and
accumulating the plurality of aligned partial products in parallel into a result queue utilizing at least one adder.

### Claim 2 (depends on 1)

The processor of claim 1, wherein the datapath is configured to convert one or more elements of the matrix operands to a half-precision, floating-point format value in a first stage of the datapath.

### Claim 3 (depends on 1)

The processor of claim 1, wherein the datapath is configured to generate a plurality of 4-vector dot products in parallel.

### Claim 4 (depends on 1)

The processor of claim 1, wherein the datapath includes a number of pipeline stages, and wherein at least one pipeline stage in the number of pipeline stages is shared with a double-precision, floating-point fused multiply accumulate (DFMA) datapath.

### Claim 5 (depends on 4)

The processor of claim 4, wherein the at least one pipeline stage comprises a completion adder that is configured to accumulate the plurality of aligned partial products in parallel into the result queue.

### Claim 6 (depends on 1)

The processor of claim 1, wherein the datapath is configured to generate the plurality of elements of the result matrix in a plurality of passes during a single instruction cycle.

### Claim 7 (depends on 1)

The processor of claim 1, wherein the processor is a parallel processing unit comprising a plurality of streaming multi-processors (SMs), each SM in the plurality of SMs including a register file and a number of cores, each core in the number of cores including an instance of the datapath.

### Claim 8 (depends on 7)

The processor of claim 7, wherein the MMA operation is configured to be executed by a number of threads in parallel, each thread configured to generate a portion of the elements in the result matrix on a particular core using different combinations of the vectors of the matrix operands specified in the instruction.

### Claim 9

A method, comprising:
receiving an instruction for a matrix multiply and accumulate (MMA) operation; and
executing, by a processor, the MMA operation to generate a plurality of elements of a result matrix at an output of a datapath,
wherein each element in the plurality of elements of the result matrix is generated by calculating at least one dot product of corresponding pairs of vectors associated with matrix operands specified in the instruction for the MMA operation,
wherein a dot product operation for calculating each dot product in the at least one dot product comprises:
generating a plurality of partial products by multiplying each element of a first vector with a corresponding element of a second vector,
aligning the plurality of partial products based on the exponents associated with each element of the first vector and each element of the second vector, and
accumulating the plurality of aligned partial products into a result queue utilizing at least one adder.

### Claim 10 (depends on 9)

The method of claim 9, wherein the datapath is configured to convert one or more elements of the matrix operands to a half-precision, floating-point format value in a first stage of the datapath.

### Claim 11 (depends on 9)

The method of claim 9, wherein the datapath is configured to generate a plurality of 4-vector dot products in parallel.

### Claim 12 (depends on 9)

The method of claim 9, wherein the datapath includes a number of pipeline stages, and wherein at least one pipeline stage in the number of pipeline stages is shared with a double-precision, floating-point fused multiply accumulate (DFMA) datapath.

### Claim 13 (depends on 9)

The method of claim 9, wherein the datapath is configured to generate the plurality of elements of the result matrix in a plurality of passes during a single instruction cycle.

### Claim 14 (depends on 9)

The method of claim 9, wherein the processor is a parallel processing unit comprising a plurality of streaming multi-processors (SMs), each SM in the plurality of SMs including a register file and a number of cores, each core in the number of cores including an instance of the datapath.

### Claim 15 (depends on 14)

The method of claim 14, wherein the MMA operation is configured to be executed by a number of threads in parallel, each thread configured to generate a portion of the elements in the result matrix on a particular core using different combinations of the vectors of the matrix operands specified in the instruction.

### Claim 16

A non-transitory, computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform steps comprising:
receiving an instruction for a matrix multiply and accumulate (MMA) operation; and
executing the MMA operation to generate a plurality of elements of a result matrix at an output of a datapath,
wherein each element in the plurality of elements of the result matrix is generated by calculating at least one dot product of corresponding pairs of vectors associated with matrix operands specified in the instruction for the MMA operation,
wherein a dot product operation for calculating each dot product in the at least one dot product comprises:
generating a plurality of partial products by multiplying each element of a first vector with a corresponding element of a second vector,
aligning the plurality of partial products based on the exponents associated with each element of the first vector and each element of the second vector, and
accumulating the plurality of aligned partial products into a result queue utilizing at least one adder.

### Claim 17 (depends on 16)

The non-transitory, computer-readable storage medium of claim 16, wherein the datapath is configured to convert one or more elements of the matrix operands to a half-precision, floating-point format value in a first stage of the datapath.

### Claim 18 (depends on 16)

The non-transitory, computer-readable storage medium of claim 16, wherein the datapath includes a number of pipeline stages, and wherein at least one pipeline stage in the number of pipeline stages is shared with a double-precision, floating-point fused multiply accumulate (DFMA) datapath.

### Claim 19 (depends on 16)

The non-transitory, computer-readable storage medium of claim 16, wherein the processor is a parallel processing unit comprising a plurality of streaming multi-processors (SMs), each SM in the plurality of SMs including a register file and a number of cores, each core in the number of cores including an instance of the datapath.

### Claim 20 (depends on 19)

The non-transitory, computer-readable storage medium of claim 19, wherein the MMA operation is configured to be executed by a number of threads in parallel, each thread configured to generate a portion of the elements in the result matrix on a particular core using different combinations of the vectors of the matrix operands specified in the instruction.
