# US 2025/0094170 A1 · Instructions and logic to perform floating point and integer operations for machine learning

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2025/0094170 A1
- Document ID: US20250094170A1
- Status: Published application
- Kind: A1
- Application number: 18/901,027
- Filed: 2024-09-30
- Published: 2025-03-20
- Earliest claimed: 2017-04-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Himanshu Kaul; Mark A. Anders; Sanu K. Mathew; Anbang Yao; Joydeep Ray; Ping T. Tang; Michael S. Strickland; Xiaoming Chen; Tatiana Shpeisman; Abhishek R. Appu; Altug Koker; Kamal Sinha; Balaji Vembu; Nicolas C. Galoppo Von Borries; Eriko Nurvitadhi; Rajkishore Barik; Tsung-Han Lin; Vasanth Ranganathan; Sanjeev Jahagirdar
- Web page: https://patentgrep.com/patent/US20250094170A1

## CPC

- G06F 9/3001: {Arithmetic instructions}
- G06F 7/483: Computations with numbers represented by a non-linear combination of denominational numbers, e.g. rational numbers, logarithmic number system or floating-point numbers {(G06F7/4806, G06F7/4824, G06F7/49, G06F7/491, G06F7/544 take precedence)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 9/30014: {with variable precision}
- G06F 9/30036: {Instructions to perform operations on packed data, e.g. vector, tile or matrix operations}
- G06F 9/3851: {from multiple instruction streams, e.g. multistreaming}
- G06F 9/3888: {controlled by a single instruction for multiple threads [SIMT] in parallel}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/063: using electronic means
- G06N 3/08: Learning methods
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G09G 5/393: Arrangements for updating the contents of the bit-mapped memory
- G06F 1/16: Constructional details or arrangements
- G06F 9/30025: {Format conversion instructions, e.g. Floating-Point to Integer, decimal conversion}
- G06F 9/3013: {according to data content, e.g. floating-point registers, address registers}
- G06F 17/16: Matrix or vector computation {, e.g. matrix-matrix or matrix-vector multiplication, matrix factorization (matrix transposition G06F7/78)}
- G06F 2207/3824: Accepting both fixed-point and floating-point numbers
- G06N 20/00: Machine learning
- G06T 15/005: {General purpose rendering architectures}
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G09G: ARRANGEMENTS OR CIRCUITS FOR CONTROL OF INDICATING DEVICES USING STATIC MEANS TO PRESENT VARIABLE INFORMATION (arrangements for transferring data between digital computers and displays G06F3/14; static indicating arrangements comprising an association of a number of separate sources or light control cells G09F9/00; static indicating arrangements comprising integral associations of a number of light sources H01J, H01K, H10H, H05B33/12; scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission, details thereof H04N1/00)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

One embodiment provides for a graphics processing unit to accelerate machine-learning operations, the graphics processing unit comprising a multiprocessor having a single instruction, multiple thread (SIMT) architecture, the multiprocessor to execute at least one single instruction; and a first compute unit included within the multiprocessor, the at least one single instruction to cause the first compute unit to perform a two-dimensional matrix multiply and accumulate operation, wherein to perform the two-dimensional matrix multiply and accumulate operation includes to compute a 32-bit intermediate product of 16-bit operands and to compute a 32-bit sum based on the 32-bit intermediate product.

## Claims

### Claim 1

An apparatus comprising:
an interconnect fabric;
a memory interface coupled to the interconnect fabric;
an input/output, IO, unit coupled to the interconnect fabric;
an array of multiprocessors coupled to the interconnect fabric, a multiprocessor in the array of multiprocessors to execute a mixed-precision instruction in parallel across multiple threads,
the multiprocessor in the array of multiprocessors comprising:
a plurality of registers to store packed floating-point operand values; and
execution circuitry to execute one or more of the mixed-precision instructions to perform a fused multiply-accumulate operation, the execution circuitry comprising:
a 16-bit multiplier to multiply a first 16-bit floating point source value and a second 16-bit floating point source value to generate an intermediate result; and
a 32-bit accumulator to add the intermediate result to an accumulated floating point value to generate a new accumulation result.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising:
a parallel processor die comprising the interconnect fabric, memory interface, the input/output (IO) unit, and the array of multiprocessors, the parallel processor die further comprising:
a local memory interconnect to couple the memory interface to stacked memory dies, the local memory interconnect comprising independent groups of memory interfaces, the independent groups of memory interfaces associated with respective memory dies of the stacked memory dies.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the mixed-precision instructions are primitives of a machine learning framework.

### Claim 4 (depends on 3)

The apparatus of claim 3, wherein the first and second 16-bit floating point source values comprise data elements a first matrix and a second matrix and wherein each of the plurality of multiplications comprises a multiplication of a data element from the first matrix and a data element from the second matrix.

### Claim 5 (depends on 4)

The apparatus of claim 4, wherein the first and second matrices are to be associated with a convolutional layer of the machine learning framework.

### Claim 6 (depends on 5)

The apparatus of claim 5, wherein the machine learning framework comprises a neural network.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein the machine learning framework comprises a recurrent neural network, RNN.

### Claim 8 (depends on 1)

The apparatus of any of claim 1, further comprising:
virtualization circuitry to share the array of multiprocessors with a plurality of virtual machines.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the virtualization circuitry comprises multiple sets of control registers to be associated with multiple corresponding virtual machines, a group of control registers to store one or more address pointers to identify a region of memory associated with a corresponding virtual machine.

### Claim 10 (depends on 2)

The apparatus of claim 2, wherein the memory interface is to couple the interconnect fabric to a memory device, the memory interface to use virtual channels to separate traffic streams to access the memory device.

### Claim 11 (depends on 1)

The apparatus of claim 1, further comprising:
a cache hierarchy to store data for the array of multiprocessors, the cache hierarchy including an L1 cache and an L2 cache to be shared between a plurality of multiprocessors within the array of multiprocessors.

### Claim 12 (depends on 2)

The apparatus of claim 2, further comprising: a memory management unit (MMU) coupled to the interconnect fabric, the MMU comprising a translation lookaside buffer (TLB) to store virtual-to-physical address translations to access the stacked memory dies.

### Claim 13 (depends on 12)

The apparatus of claim 12, wherein the MMU is to use a shared virtual memory address space to access the stacked memory dies and one or more system memory devices.

### Claim 14 (depends on 2)

The apparatus of claim 2, wherein the stacked memory dies comprise a High Bandwidth Memory, HBM, device.

### Claim 15

An apparatus comprising:
an interconnect fabric;
a memory interface coupled to the interconnect fabric;
an input/output, IO, unit coupled to the interconnect fabric;
an array of multiprocessors coupled to the interconnect fabric, a multiprocessor in the array of multiprocessors comprising:
a plurality of registers to store packed floating-point and packed integer operand values including 32-bit floating-point values, 16-bit floating-point values, and 8-bit integer values; and
a decoder to decode a plurality of mixed-precision fused multiply-accumulate (FMA) instructions including a first FMA instruction indicating N 16-bit floating-point source operands and a 32-bit floating-point source operand, and a second FMA instruction indicating 2N 8-bit integer source operands and a 32-bit integer source operand, and
parallel multiplication circuitry to:
perform N/2 parallel 16-bit floating-point multiplications responsive to the first FMA instruction to produce N/2 floating-point products;
perform N parallel 8-bit integer multiplications responsive to the second FMA instruction to produce N integer products; and
accumulation circuitry to:
add the N/2 floating point products to the 32-bit floating-point source operand responsive to the first FMA instruction to generate an accumulated 32-bit floating-point result; and
add the N integer products to the 32-bit integer source operand responsive to the second FMA instruction to generate an accumulated 32-bit integer result.

### Claim 16 (depends on 15)

The apparatus of claim 15, wherein the N/2 parallel 16-bit floating-point or N parallel 8-bit integer multiplications are performed in a single clock cycle.

### Claim 17 (depends on 15)

The apparatus of claim 15, wherein the decoder is to further decode a third FMA instruction indicating N/2 32-bit floating point source operands, the parallel multiplication circuitry, responsive to the third FMA instruction, to further perform N/4 single-precision floating point multiplications to generate N/4 product(s).

### Claim 18 (depends on 15)

The apparatus of claim 15, wherein the 32-bit floating-point source operand and the 32-bit integer source operand comprise accumulated values from one or more prior instances of the first FMA instruction and the second FMA instruction, respectively.

### Claim 19 (depends on 15)

The apparatus of claim 15, further comprising:
a parallel processor die comprising the interconnect fabric, memory interface, the input/output (IO) unit, and the array of multiprocessors, the parallel processor die further comprising:
a local memory interconnect to couple the memory interface to stacked memory dies, the local memory interconnect comprising independent groups of memory interfaces, the independent groups of memory interfaces associated with respective memory dies of the stacked memory dies.

### Claim 20 (depends on 15)

The apparatus of claim 15, wherein the mixed-precision instructions are primitives of a machine learning framework.

### Claim at position 21

21-31. (canceled)