Quantization-aware training with numerical overflow avoidance for neural networks

A method, comprising:

• storing, by circuitry of a memory, a plurality of input data values for a neural network;
• selecting, by circuitry of a processor, a first weight value with a first magnitude, based at least in part on an input data value used for training a node of a neural network;
• and
• replacing, by the circuitry, the first weight value with a second weight value that has a second magnitude different from the first magnitude, based at least in part on the first magnitude and one or more magnitudes of a first set of weight values comprising one or more weight values generated previously for the node;
• and
• generating, by the circuitry, an output value for the node based at least in part on the second magnitude.

A computing system comprising:

• a memory comprising circuitry configured to store a plurality of input data values for training a neural network;
• a processor comprising circuitry configured to:
• select a first weight value with a first magnitude, based at least in part on an input data value of the plurality of input data values used for training a node of a neural network;
• replace the first weight value with a second weight value that has a second magnitude different from the first magnitude, based at least in part on the first magnitude and one or more magnitudes of a first set of weight values comprising one or more weight values generated previously for the node;
• and
• generate an output value for the node based at least in part on the second magnitude.