# US 12,217,053 B2 · Instructions and logic to perform floating point and integer operations for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,053 B2
- Document ID: US12217053B2
- Status: Granted
- Kind: B2
- Application number: 18/528,340
- Filed: 2023-12-04
- Published: 2024-06-06
- Granted: 2025-02-04
- Earliest claimed: 2017-04-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Himanshu Kaul; Mark A. Anders; Sanu K. Mathew; Anbang Yao; Joydeep Ray; Ping T. Tang; Michael S. Strickland; Xiaoming Chen; Tatiana Shpeisman; Abhishek R. Appu; Altug Koker; Kamal Sinha; Balaji Vembu; Nicolas C. Galoppo Von Borries; Eriko Nurvitadhi; Rajkishore Barik; Tsung-Han Lin; Vasanth Ranganathan; Sanjeev Jahagirdar
- Web page: https://patentgrep.com/patent/US12217053B2

## CPC

- G06F 9/3001: {Arithmetic instructions}
- G06F 7/483: Computations with numbers represented by a non-linear combination of denominational numbers, e.g. rational numbers, logarithmic number system or floating-point numbers {(G06F7/4806, G06F7/4824, G06F7/49, G06F7/491, G06F7/544 take precedence)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 9/30014: {with variable precision}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3851: {from multiple instruction streams, e.g. multistreaming}
- G06F 9/3888: {controlled by a single instruction for multiple threads [SIMT] in parallel}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G09G 5/393: Arrangements for updating the contents of the bit-mapped memory
- G06F 1/16: Constructional details or arrangements
- G06F 9/30025: {Format conversion instructions, e.g. Floating-Point to Integer, decimal conversion}
- G06F 9/3013: {according to data content, e.g. floating-point registers, address registers}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 2207/3824: Accepting both fixed-point and floating-point numbers
- G06N 20/00: Machine learning
- G06T 15/005: {General purpose rendering architectures}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

One embodiment provides for a graphics processing unit to accelerate machine-learning operations, the graphics processing unit comprising a multiprocessor having a single instruction, multiple thread (SIMT) architecture, the multiprocessor to execute at least one single instruction; and a first compute unit included within the multiprocessor, the at least one single instruction to cause the first compute unit to perform a two-dimensional matrix multiply and accumulate operation, wherein to perform the two-dimensional matrix multiply and accumulate operation includes to compute an intermediate product of 16-bit operands and to compute a 32-bit sum based on the intermediate product.

## Claims

### Claim 1

An apparatus comprising:
an array of processing units to process instructions;
a plurality of memory interfaces, each memory interface to couple one or more of the processing units to a corresponding memory;
each processing unit in the array of processing units comprising:
execution circuitry to execute a floating-point multiply-accumulate instruction, the execution circuitry comprising:
mantissa multiplication circuitry, wherein the mantissa multiplication circuitry is shared with an integer datapath of the execution circuitry, wherein responsive to the floating-point multiply-accumulate instruction, the mantissa multiplication circuitry is to perform a multiplication operation with a mantissa value of each 16-bit floating-point data element of a first plurality of 16-bit floating-point data elements and a mantissa value of a corresponding 16-bit floating-point data element of a second plurality of 16-bit floating-point data elements to generate a corresponding plurality of mantissa results;
exponent processing circuitry, responsive to the floating-point multiply-accumulate instruction, to perform an operation with an exponent value of each 16-bit floating-point data element of the first plurality of 16-bit floating-point data elements and an exponent value of each corresponding 16-bit floating-point data element of the second plurality of 16-bit floating-point data elements to generate a corresponding plurality of exponent results;
circuitry to process the plurality of mantissa results and the plurality of exponent results to generate a corresponding floating-point product; and
adder circuitry to generate a plurality of result floating-point values, each result floating-point value comprising a sum of one or more floating-point products of a plurality of floating-point products and a corresponding accumulated floating-point value of a plurality of accumulated floating-point values.

### Claim 2 (depends on 1)

The apparatus of claim 1 wherein each 16-bit floating-point data element of the first plurality of 16-bit floating-point data elements and the second plurality of 16-bit floating-point data elements includes a 1-bit sign, a 7-bit mantissa value and an 8-bit exponent value.

### Claim 3 (depends on 2)

The apparatus of claim 2 further comprising:
hardware logic to convert a first plurality of 32-bit floating-point data elements into the first plurality of 16-bit floating-point data elements and to convert a second plurality of 32-bit floating-point data elements into the second plurality of 16-bit floating-point data elements.

### Claim 4 (depends on 1)

The apparatus of claim 1 wherein the circuitry to process each mantissa result of the plurality of mantissa results comprises shift circuitry to shift one or more of the plurality of mantissa results.

### Claim 5 (depends on 4)

The apparatus of claim 4 further comprising:
normalization circuitry to normalize one or more of the plurality of result floating-point values.

### Claim 6 (depends on 5)

The apparatus of claim 5 wherein each result floating-point value of the plurality of result floating-point values comprise a 32-bit floating-point value.

### Claim 7 (depends on 1)

The apparatus of claim 1 further comprising:
a register set to store the first plurality of 16-bit floating-point data elements, the second plurality of 16-bit floating-point data elements, the plurality of result floating-point values, and the plurality of accumulated floating-point values.

### Claim 8 (depends on 1)

The apparatus of claim 1 wherein the multiplication operation comprises a portion of multiply-accumulate operations of a convolutional neural network layer.

### Claim 9 (depends on 8)

The apparatus of claim 8 wherein the first plurality of 16-bit floating-point data elements comprise weights associated with the convolutional neural network layer.

### Claim 10 (depends on 9)

The apparatus of claim 9 wherein a corresponding result floating-point data element comprises an activation value.

### Claim 11 (depends on 1)

The apparatus of claim 1 further comprising:
a processor coupled to the array of processing units, the processor comprising a plurality of cores.

### Claim 12

A system comprising:
a host processor comprising a plurality of cores, at least one core of the plurality of cores to execute one or more applications;
a memory interface to couple the host processor to a system memory; and
an interconnect to couple the host processor to an accelerator, the accelerator comprising:
an array of processing units to process instructions;
a plurality of memory interfaces, each memory interface to couple one or more of the processing units to a corresponding memory of a plurality of memories;
each processing unit in the array of processing units comprising:
execution circuitry to execute a floating-point multiply-accumulate instruction, the execution circuitry comprising:
mantissa multiplication circuitry, wherein the mantissa multiplication circuitry is shared with an integer datapath of the execution circuitry, wherein responsive to the floating-point multiply-accumulate instruction, the mantissa multiplication circuitry is to perform a multiplication operation with a mantissa value of each 16-bit floating-point data element of a first plurality of 16-bit floating-point data elements and a mantissa value of a corresponding 16-bit floating-point data element of a second plurality of 16-bit floating-point data elements to generate a corresponding plurality of mantissa results;
exponent processing circuitry, responsive to the floating-point multiply-accumulate instruction, to perform an operation with an exponent value of each 16-bit floating-point data element of the first plurality of 16-bit floating-point data elements and an exponent value of each corresponding 16-bit floating-point data element of the second plurality of 16-bit floating-point data elements to generate a corresponding plurality of exponent results;
circuitry to process the plurality of mantissa results and the plurality of exponent results to generate a corresponding floating-point product; and
adder circuitry to generate a plurality of result floating-point values, each result floating-point value comprising a sum of one or more floating-point products of a plurality of floating-point products and a corresponding accumulated floating-point value of a plurality of accumulated floating-point values.

### Claim 13 (depends on 12)

The system of claim 12 wherein each core of the plurality of cores comprises:
a data cache to store data;
a translation lookaside buffer (TLB) to store address translations; and
execution circuitry to execute instructions of the one or more applications.

### Claim 14 (depends on 12)

The system of claim 12 further comprising:
coherency circuitry to maintain coherency of data stored in host memory and one or more of the plurality of memories.

### Claim 15 (depends on 12)

The system of claim 12 wherein each 16-bit floating-point data elements of the first plurality of 16-bit floating-point data elements and the second plurality of 16-bit floating-point data elements includes a 1-bit sign, a 7-bit mantissa value and an 8-bit exponent value.

### Claim 16 (depends on 15)

The system of claim 15 further comprising:
hardware logic to convert a first plurality of 32-bit floating-point data elements into the first plurality of 16-bit floating-point data elements and to convert a second plurality of 32-bit floating-point data elements into the second plurality of 16-bit floating-point data elements.

### Claim 17 (depends on 12)

The system of claim 12 wherein the circuitry to process each mantissa result of the plurality of mantissa results comprises shift circuitry to shift one or more of the plurality of mantissa results.

### Claim 18 (depends on 17)

The system of claim 17 further comprising:
normalization circuitry to normalize one or more of the plurality of result floating-point values.

### Claim 19 (depends on 18)

The system of claim 18 wherein each result floating-point value of the plurality of result floating-point values comprise a 32-bit floating-point value.

### Claim 20 (depends on 12)

The system of claim 12 further comprising:
a register set to store the first plurality of 16-bit floating-point data elements, the second plurality of 16-bit floating-point data elements, the plurality of result floating-point values, and the plurality of accumulated floating-point values.

### Claim 21 (depends on 12)

The system of claim 12 wherein the multiplication operation comprises a portion of multiply-accumulate operations of a convolutional neural network layer.

### Claim 22 (depends on 21)

The system of claim 21 wherein the first plurality of 16-bit floating-point data elements comprise weights associated with the convolutional neural network layer.

### Claim 23 (depends on 22)

The system of claim 22 wherein a corresponding result floating-point data element comprises an activation value.

### Claim 24 (depends on 12)

The system of claim 12 further comprising:
a processor coupled to the array of processing units, the processor comprising a plurality of cores.

### Claim 25

A method for performing a floating-point multiply-accumulate operation on a plurality of floating-point data elements, the method comprising:
multiplying mantissas of each 16-bit floating-point data element of a first plurality of 16-bit floating-point data elements by a corresponding mantissa value of a corresponding 16-bit floating-point data element of a second plurality of 16-bit floating-point data elements using mantissa multiplication circuitry shared with an integer datapath, to generate a corresponding plurality of mantissa results;
performing an operation with an exponent value of each 16-bit floating-point data element of the first plurality of 16-bit floating-point data elements and an exponent value of each corresponding 16-bit floating-point data element of the second plurality of 16-bit floating-point data elements to generate a corresponding plurality of exponent results;
processing the plurality of mantissa results and the plurality of exponent results to generate a corresponding floating-point product; and
generating a plurality of result floating-point values, each result floating-point value comprising a sum of one or more floating-point products of a plurality of floating-point products and a corresponding accumulated floating-point value of a plurality of accumulated floating-point values.