Incremental precision networks using residual inference and fine-grain quantization
AssigneeIntel Corporation
InventorsAbhisek Kundu, Naveen Mellempudi, Dheevatsa Mudigere, Dipankar Das
Life of this patent
Estimating the term…
Abstract
One embodiment provides for a computer-readable medium storing instructions that cause one or more processors to perform operations comprising determining a per-layer scale factor to apply to tensor data associated with layers of a neural network model and converting the tensor data to converted tensor data. The tensor data may be converted from a floating point datatype to a second datatype that is an 8-bit datatype. The instructions further cause the one or more processors to generate an output tensor based on the converted tensor data and the per-layer scale factor.
Claims25
A non-transitory computer-readable medium storing instructions, execution of which by one or more processors including a general-purpose graphics processor, causes the one or more processors to perform operations comprising:
- determining a first scale factor to apply to first tensor data associated with a first layer of a neural network model;
- converting the first tensor data associated with the first layer of the neural network model to generate first converted tensor data, the first tensor data converted from a first datatype to a second datatype, wherein the second datatype is an 8-bit datatype;
- generating a first output tensor that is associated with the first layer of the neural network model, the first output tensor generated via the general-purpose graphics processor based on the first converted tensor data and the first scale factor;
- determining a second scale factor to apply to second tensor data associated with a second layer of the neural network model;
- converting the second tensor data associated with the second layer of the neural network model to generate second converted tensor data, the second tensor data converted from the first datatype to the second datatype;
- and
- generating a second output tensor that is associated with the second layer of the neural network model, the second output tensor generated via the general-purpose graphics processor based on the second converted tensor data and the second scale factor.
An apparatus comprising:
- a memory device;
- and
- one or more processors coupled with the memory device, the one or more processors including a general-purpose graphics processor, the one or more processors comprising circuitry configured to:
- determine a first scale factor to apply to first tensor data associated with a first layer of a neural network model;
- convert the first tensor data associated with the first layer of the neural network model to generate first converted tensor data, the first tensor data converted from a first datatype to a second datatype, wherein the second datatype is an 8-bit datatype;
- generate a first output tensor that is associated with the first layer of the neural network model, the first output tensor generated via the general-purpose graphics processor based on the first converted tensor data and the first scale factor;
- determine a second scale factor to apply to second tensor data associated with a second layer of the neural network model;
- convert the second tensor data associated with the second layer of the neural network model to generate second converted tensor data, the second tensor data converted from the first datatype to the second datatype;
- and
- generate a second output tensor that is associated with the second layer of the neural network model, the second output tensor generated via the general-purpose graphics processor based on the second converted tensor data and the second scale factor.
A general-purpose graphics processor comprising:
- a system interface;
- and
- a machine learning accelerator circuitry coupled with the system interface, the machine learning accelerator circuitry configured to:
- convert first tensor data associated with a first layer of a neural network model to generate first converted tensor data, the first tensor data converted from a first datatype to a second datatype, wherein the second datatype is an 8-bit datatype;
- generate a first output tensor that is associated with the first layer of the neural network model, the first output tensor generated via the general-purpose graphics processor based on the first converted tensor data and a first scale factor, the first scale factor determined for the first tensor data;
- convert second tensor data associated with a second layer of the neural network model to generate second converted tensor data, the second tensor data converted from the first datatype to the second datatype;
- and
- generate a second output tensor that is associated with the second layer of the neural network model, the second output tensor generated via the general-purpose graphics processor based on the second converted tensor data and a second scale factor, the second scale factor determined for the second tensor data.
Dependencies are inferred from the claim text.