# US 11,308,574 B2 · Compute optimizations for low precision machine learning operations

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,574 B2
- Document ID: US11308574B2
- Status: Granted
- Kind: B2
- Application number: 16/983,080
- Filed: 2020-08-03
- Published: 2020-11-19
- Granted: 2022-04-19
- Earliest claimed: 2017-04-28
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: Elmoustapha Ould-Ahmed-Vall; Sara S. Baghsorkhi; Anbang Yao; Kevin Nealis; Xiaoming Chen; Altug Koker; Abhishek R. Appu; John C. Weast; Mike B. Macpherson; Dukhwan Kim; Linda L. Hurd; Ben J. Ashbaugh; Barath Lakshmanan; Liwei Ma; Joydeep Ray; Ping T. Tang; Michael S. Strickland
- Web page: https://patentgrep.com/patent/US11308574B2

## CPC

- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06F 7/483: Computations with numbers represented by a non-linear combination of denominational numbers, e.g. rational numbers, logarithmic number system or floating-point numbers {(G06F7/4806, G06F7/4824, G06F7/49, G06F7/491, G06F7/544 take precedence)}
- G06F 9/30014: {with variable precision}
- G06F 9/30185: {according to one or more bits in the instruction, e.g. prefix, sub-opcode}
- G06F 9/3863: {using multiple copies of the architectural state, e.g. shadow registers}
- G06F 9/5044: {considering hardware capabilities}
- G06F 12/0811: with multilevel cache hierarchies
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/063: using electronic means
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06N 20/00: Machine learning
- G06F 3/14: Digital output to display device {; Cooperation and interconnection of the display device with other functional units}
- G06F 2212/401: Compressed data
- G06T 1/60: Memory management
- G06T 15/005: {General purpose rendering architectures}
- Y02D 10/00: Energy efficient computing, e.g. low power processors, power management or thermal management
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- Y02D: CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE

## Abstract

Embodiments described herein provide a graphics processor that can perform a variety of mixed and multiple precision instructions and operations. One embodiment provides a streaming multiprocessor that can concurrently execute multiple thread groups, wherein the streaming multiprocessor includes a single instruction, multiple thread (SIMT) architecture and the streaming multiprocessor is to execute multiple threads for each of multiple instructions. The streaming multiprocessor can perform concurrent integer and floating-point operations and includes a mixed precision core to perform operations at multiple precisions.

## Claims

### Claim 1

An apparatus comprising:
a semiconductor substrate;
a 3D memory stack comprising a plurality of stacked memory dies integrated on the semiconductor substrate;
a parallel processor die mounted on the semiconductor substrate;
a local memory interconnect to couple the parallel processor die to the 3D memory stack, the local memory interconnect comprising a plurality of memory interfaces, each memory interface associated with a memory die of the plurality of stacked memory dies;
wherein the parallel processor die comprises:
an interconnect fabric comprising one or more crossbar switches;
a memory controller coupled to the local memory interconnect via the memory interfaces to provide access to the 3D memory stack, the memory controller coupled to the interconnect fabric;
an input/output (TO) interface coupled to the interconnect fabric; and
an array of graphics processor compute units coupled to the interconnect fabric to process mixed-precision dot-product instructions, at least one graphics processor compute unit comprising:
a plurality of packed data registers to store a plurality of packed data elements at a first precision; and
mixed-precision execution circuitry to execute the mixed-precision dot-product instructions, the mixed-precision execution circuitry to perform a plurality of multiplications of different pairs of the plurality of packed data elements to generate a corresponding plurality of products and to add the corresponding plurality of products to an accumulation value stored at a second precision greater than the first precision to generate a result at the second precision.

### Claim 2 (depends on 1)

The apparatus of claim 1 wherein at least some of the plurality of packed data elements comprise data elements of matrices.

### Claim 3 (depends on 2)

The apparatus of claim 2 wherein the mixed-precision dot-product instructions are primitives of a machine learning framework.

### Claim 4 (depends on 3)

The apparatus of claim 3 wherein the matrices are associated with a convolutional layer of the machine-learning framework.

### Claim 5 (depends on 4)

The apparatus of claim 4 wherein the matrices associated with the convolutional layer of the machine-learning framework comprise a first matrix and a second matrix, each of the plurality of multiplications comprises a multiplication of a packed data element from the first matrix and a packed data element from the second matrix, the mixed-precision execution circuitry is to generate an output matrix comprising data elements generated by a multiplication of each packed data element from the first matrix and each packed data element from the second matrix, the mixed-precision execution circuitry is to evaluate an activation function based on the output matrix, and the activation function is a primitive of the machine learning framework.

### Claim 6 (depends on 3)

The apparatus of claim 3 wherein the machine learning framework comprises a neural network.

### Claim 7 (depends on 6)

The apparatus of claim 6 wherein the neural network comprises a recurrent neural network (RNN).

### Claim 8 (depends on 1)

The apparatus of claim 1 further comprising:
virtualization circuitry to share the array of graphics processor compute units with a plurality of virtual machines.

### Claim 9 (depends on 8)

The apparatus of claim 8 wherein the virtualization circuitry comprises multiple sets of control registers to be associated with multiple corresponding virtual machines, a group of control registers to store one or more address pointers to identify a region of memory associated with a corresponding virtual machine.

### Claim 10 (depends on 1)

The apparatus of claim 1 wherein a memory interface comprises a physical memory channel, and wherein one or more virtual memory channels are to be associated with a physical memory channel.

### Claim 11 (depends on 1)

The apparatus of claim 1 further comprising:
a cache hierarchy to store data for the array of graphics processor compute units, the cache hierarchy including an L1 cache and an L2 cache to be shared between the of array of graphics processor compute units.

### Claim 12 (depends on 1)

The apparatus of claim 1 further comprising:
memory management circuitry to map physical addresses of the 3D memory stack to a shared virtual memory space and to access the physical addresses using shared virtual memory (SVM) technology.

### Claim 13 (depends on 1)

The apparatus of claim 1 further comprising:
an input/output memory management unit (IOMMU) coupled to the interconnect fabric, the IOMMU comprising a translation buffer to store virtual-to-physical address translations to access the 3D memory stack.

### Claim 14 (depends on 13)

The apparatus of claim 13 wherein a first one or more virtual-to-physical address translations are to identify regions in the 3D memory stack and wherein a second one or more virtual-to-physical address translations are to identify regions in a system memory device.

### Claim 15 (depends on 14)

The apparatus of claim 14 wherein the array of graphics processor compute units are to connect to the system memory device via the IO interface.

### Claim 16 (depends on 1)

The apparatus of claim 1, wherein the 3D memory stack comprises a High Bandwidth Memory (HBM) memory device.

### Claim 17

A graphics processor comprising:
a semiconductor substrate;
a parallel processor die mounted on the semiconductor substrate, the parallel processor die comprising:
an interconnect fabric comprising one or more crossbar switches;
an input/output (TO) interface coupled to the interconnect fabric; and
an array of graphics processor compute units coupled to the interconnect fabric to process mixed-precision dot-product instructions, at least one graphics processor compute unit comprising:
a plurality of packed data registers to store a plurality of packed data elements at a first precision; and
mixed-precision execution circuitry to execute the mixed-precision dot-product instructions, the mixed-precision execution circuitry to perform a plurality of multiplications of different pairs of the plurality of packed data elements to generate a corresponding plurality of products and to add the corresponding plurality of products to an accumulation value stored at a second precision greater than the first precision to generate a result at the second precision.

### Claim 18 (depends on 17)

The graphics processor of claim 17 further comprising:
a 3D memory stack comprising a plurality of stacked memory dies integrated on the semiconductor substrate;
a local memory interconnect to couple the parallel processor die to the 3D memory stack, the local memory interconnect comprising a plurality of memory interfaces, each memory interface associated with a memory die of the plurality of stacked memory dies; and
a memory controller coupled to the local memory interconnect via the memory interfaces to provide access to the 3D memory stack, the memory controller coupled to the interconnect fabric.

### Claim 19 (depends on 18)

The graphics processor of claim 18 wherein a memory interface comprises a physical memory channel and one or more virtual memory channels are to be associated with a physical memory channel, and the graphics processor further comprises:
a cache hierarchy to store data for the array of graphics processor compute units, the cache hierarchy including an L1 cache and an L2 cache to be shared between the array of graphics processor compute units;
memory management circuitry to map physical addresses of the 3D memory stack to a shared virtual memory space and to access the physical addresses using shared virtual memory (SVM) technology; and
an input/output memory management unit (IOMMU) coupled to the interconnect fabric, the IOMMU comprising a translation buffer to store virtual-to-physical address translations to access the 3D memory stack.

### Claim 20 (depends on 19)

The graphics processor of claim 19 wherein a first one or more virtual-to-physical address translations are to identify regions in the 3D memory stack and wherein a second one or more virtual-to-physical address translations are to identify regions in a system memory device, wherein the array of graphics processor compute units are to connect to the system memory device via the IO interface, and wherein the 3D memory stack comprises a High Bandwidth Memory (HBM) memory device.

### Claim 21 (depends on 17)

The graphics processor of claim 17 wherein at least some of the plurality of packed data elements comprise data elements of matrices, the mixed-precision dot-product instructions are primitives of a machine learning framework, and the matrices are associated with a convolutional layer of the machine-learning framework.

### Claim 22 (depends on 21)

The graphics processor of claim 21 wherein the matrices associated with the convolutional layer of the machine-learning framework comprise a first matrix and a second matrix, each of the plurality of multiplications comprises a multiplication of a packed data element from the first matrix and a packed data element from the second matrix, the mixed-precision execution circuitry is to generate an output matrix comprising data elements generated by a multiplication of each packed data element from the first matrix and each packed data element from the second matrix, the mixed-precision execution circuitry is to evaluate an activation function based on the output matrix, and the activation function is a primitive of the machine learning framework.

### Claim 23 (depends on 21)

The graphics processor of claim 21 wherein the machine learning framework comprises a neural network and the neural network comprises a recurrent neural network (RNN).

### Claim 24 (depends on 17)

The graphics processor of claim 17 further comprising:
virtualization circuitry to share the array of graphics processor compute units with a plurality of virtual machines, wherein the virtualization circuitry comprises multiple sets of control registers to be associated with multiple corresponding virtual machines, a group of control registers to store one or more address pointers to identify a region of memory associated with a corresponding virtual machine.