# US 2018/0322382 A1 · Scaling half-precision floating point tensors for training deep neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2018/0322382 A1
- Document ID: US20180322382A1
- Status: Published application
- Kind: A1
- Application number: 15/869,582
- Filed: 2018-01-12
- Published: 2018-11-08
- Earliest claimed: 2017-05-03
- Assignees: Intel Corporation
- Applicants: Intel Corporation
- Inventors: NAVEEN MELLEMPUDI; DIPANKAR DAS
- Web page: https://patentgrep.com/patent/US20180322382A1

## CPC

- G06N 3/063: using electronic means
- G06F 5/012: {in floating-point computations}
- G06F 7/487: Multiplying; Dividing {(G06F7/4833, G06F7/4836 take precedence)}
- G06F 7/5443: {Sum of products (for applications thereof, see the relevant places, e.g. G06F17/10, H03H17/00)}
- G06F 9/30014: {with variable precision}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/0895: Weakly supervised learning, e.g. semi-supervised or self-supervised learning
- G06N 3/09: Supervised learning
- G06N 3/098: Distributed learning, e.g. federated learning
- G06T 1/20: Processor architectures; Processor configuration, e.g. pipelining
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL

## Abstract

One embodiment provides for a machine-learning accelerator device a multiprocessor to execute parallel threads of an instruction stream, the multiprocessor including a compute unit, the compute unit including a set of functional units, each functional unit to execute at least one of the parallel threads of the instruction stream. The compute unit includes compute logic configured to execute a single instruction to scale an input tensor associated with a layer of a neural network according to a scale factor, the input tensor stored in a floating-point data type, the compute logic to scale the input tensor to enable a data distribution of data of the input tensor to be represented by a 16-bit floating point data type.

## Claims

### Claim 1

A machine-learning accelerator comprising:
a multiprocessor to execute parallel threads of an instruction stream, the multiprocessor including a compute unit, the compute unit including a set of functional units, each functional unit to execute at least one of the parallel threads of the instruction stream; and
wherein the compute unit includes compute logic configured to execute a single instruction to scale an input tensor associated with a layer of a neural network according to a scale factor, the input tensor stored in a floating-point data type, the compute logic to scale the input tensor to enable a data distribution of data of the input tensor to be represented by a 16-bit floating point data type.

### Claim 2 (depends on 1)

The machine-learning accelerator as in claim 1, the compute logic to compute an exponent bias based on an absolute maximum value of the input tensor and a dynamic range of the 16-bit floating point data type.

### Claim 3 (depends on 2)

The machine-learning accelerator as in claim 2, the compute logic to convert weights and activations of the layer of the neural network to scaled 16-bit floating-point tensors.

### Claim 4 (depends on 3)

The machine-learning accelerator as in claim 3, the compute logic to perform a compute operation on the scaled 16-bit floating-point tensors, the compute operation including one a multiply, add, or a fused multiply-add.

### Claim 5 (depends on 4)

The machine-learning accelerator as in claim 4, the compute logic to generate a set of 32-bit intermediate results in response to the compute operation on the scaled 16-bit floating-point tensors.

### Claim 6 (depends on 5)

The machine-learning accelerator as in claim 5, the compute logic to re-scale the set of 32-bit intermediate results using the exponent bias and down-convert the set of 32-bit intermediate results into scaled 16-bit floating point values.

### Claim 7 (depends on 1)

The machine-learning accelerator as in claim 1, the compute logic to perform a compute operation on scaled 16-bit floating-point tensors to generate intermediate values, determine if the intermediate values are close to saturation, and re-scale the intermediate values when the intermediate values are close to saturation.

### Claim 8 (depends on 7)

The machine-learning accelerator as in claim 7, the compute logic to update an exponent bias value after the re-scale of the intermediate values.

### Claim 9

A method implemented via a machine-learning accelerator, the method comprising:
executing parallel threads of a parallel instruction stream, the parallel instruction stream including at least one single instruction to perform a scaled tensor compute operation;
in response to the single instruction, scaling data of an input tensor associated with a layer of a neural network according to a scale factor, executing one or more compute operations on scaled data of the input tensor, and re-scaling computed data of the compute operations to generate re-scaled computed data;
down-converting the re-scaled computed data; and
storing down-converted re-scaled computed data to a 16-bit floating-point data type.

### Claim 10 (depends on 9)

The method as in claim 9, additionally comprising computing an exponent bias based on an absolute maximum value of the input tensor and a dynamic range of the 16-bit floating-point data type and scaling the data of the input tensor using the exponent bias.

### Claim 11 (depends on 10)

The method as in claim 10, additionally comprising converting weights and activations of the layer of the neural network to scaled 16-bit floating-point tensors.

### Claim 12 (depends on 11)

The method as in claim 11, wherein executing the one or more compute operations on the scaled data of the input tensor includes executing a multiply, add, or a fused multiply-add operation.

### Claim 13 (depends on 12)

The method as in claim 12, wherein computed data of the compute operations include a set of 32-bit intermediate results.

### Claim 14 (depends on 13)

The method as in claim 13, additionally comprising re-scaling the set of 32-bit intermediate results using the exponent bias and down-converting the set of 32-bit intermediate results into scaled 16-bit floating-point values.

### Claim 15 (depends on 9)

The method as in claim 9, additionally comprising:
performing a compute operation on scaled 16-bit floating-point tensors to generate intermediate values;
determining if the intermediate values are close to saturation;
re-scaling the intermediate values in response to determining that the intermediate values are close to saturation; and
updating an exponent bias value after re-scaling the intermediate values.

### Claim 16

A data processing system comprising:
a non-transitory machine-readable medium to store instructions for execution by one or more processors of the data processing system; and
a general-purpose graphics processing unit including instruction decode logic to decode a single instruction including multiple operands into a single decoded instruction, at least one of the multiple operands associated with input tensor data of a layer of a neural network, and a compute unit including compute logic configured to execute the single instruction to scale data of the input tensor to enable a data distribution of data of the input tensor to be represented by a 16-bit floating point data type.

### Claim 17 (depends on 16)

The data processing system as in claim 16, the compute logic to:
compute an exponent bias based on an absolute maximum value of the input tensor and a dynamic range of the 16-bit floating point data type;
convert weights and activations of the layer of the neural network to scaled 16-bit floating-point tensors;
perform a compute operation on the scaled 16-bit floating-point tensors;
generate a set of 32-bit intermediate results in response to the compute operation
on the scaled 16-bit floating-point tensors;
re-scale the set of 32-bit intermediate results using the exponent bias; and
down-convert the set of 32-bit intermediate results into scaled 16-bit floating point values.

### Claim 18 (depends on 17)

The data processing system as in claim 17, the compute operation including one a multiply, add, or a fused multiply-add.

### Claim 19 (depends on 16)

The data processing system as in claim 16, the compute logic to perform a compute operation on scaled 16-bit floating-point tensors to generate intermediate values, determine if the intermediate values are close to saturation, and re-scale the intermediate values when the intermediate values are close to saturation.

### Claim 20 (depends on 19)

The data processing system as in claim 19, the compute logic to update an exponent bias value after the re-scale of the intermediate values.