Neural network circuitry having floating point format with asymmetric range

A method comprising:

• utilizing floating point unit circuitry to perform at least one iteration of a neural network training process, the at least one iteration comprising forward propagation neural network operations and backward propagation neural network operations, wherein utilizing the floating point unit circuitry to perform the at least one iteration of the neural network training process comprises:
• selectively configuring the floating point unit circuitry during the neural network training process to operate on floating point numbers with different n-bit floating point formats for the forward propagation neural network operations and the backward propagation neural network operations, the different n-bit floating point formats having a same number n of bits, but different formats with respect to exponent bits and mantissa bits, wherein in selectively configuring the floating point unit circuitry comprises:
• selectively configuring the floating point unit circuitry during the neural network training process to operate using a first n-bit floating point format to perform the forward propagation neural network operations on floating point numbers having the first n-bit floating point format, the first n-bit floating point format having a configuration consisting of a sign bit, m exponent bits and p mantissa bits where m is greater than p;
• selectively configuring the floating point unit circuitry during the neural network training process to operate using a second n-bit floating point format to perform the backward propagation neural network operations on floating point numbers having the second n-bit floating point format that is different than the first n-bit floating point format, the second n-bit floating point format having a configuration consisting of a sign bit, q exponent bits and r mantissa bits where q is greater than m and r is less than p;
• and
• selectively converting computation results, which are generated and output from the floating point unit circuitry as a result of performing the forward and backward propagation neural network operations, into one of the first n-bit floating point format and the second n-bit floating point format for further processing by the floating point unit circuitry in a next iteration of the neural network training process.

An apparatus, comprising:

• control circuitry for a neural network, the control circuitry comprising floating point unit circuitry, wherein the control circuitry is configured to:
• configure the floating point unit circuitry to perform forward propagation neural network operations on floating point numbers of activation values and weight values having a first n-bit floating point format, wherein n is 8 or less, wherein the first n-bit floating point format comprises a first plurality of exponent bits and a first exponent bias for the first plurality of exponent bits, wherein the forward propagation neural network operations comprise multiplying n-bit floating point activation values and n-bit floating point weight values to compute 2n-bit floating point accumulated activation values, and performing activation function operations on the 2n-bit floating point accumulated activation values to compute 4n-bit floating point output values;
• perform a conversion process to convert the 4n-bit floating point output values to n-bit floating point values, which have the first n-bit floating point format, for subsequent processing by the floating point unit circuitry, and monitor the conversion process for an occurrence of an overflow event;
• automatically adjust the first exponent bias of the first n-bit floating point format, in response to the occurrence of an overflow event during the conversion process to prevent overflow;
• configure the floating point unit circuitry to utilize the first n-bit floating point format with the adjusted first exponent bias for a next iteration of the forward propagation neural network operations;
• and
• configure the floating point unit circuitry to perform backward propagation neural network operations on floating point numbers of error gradient values having a second n-bit floating point format, which comprises a same total number of bits as the first n-bit floating point format, and a second plurality of exponent bits having a number of exponent bits which differs from a number of exponent bits of the first plurality of exponent bits.

A method, comprising:

• configuring, by control circuitry for a neural network, floating point unit circuitry to perform forward propagation neural network operations on floating point numbers of activation values and weight values having a first n-bit floating point format, wherein n is 8 or less, wherein the first n-bit floating point format comprises a first plurality of exponent bits and a first exponent bias for the first plurality of exponent bits, wherein the forward propagation neural network operations comprise multiplying n-bit floating point activation values and n-bit floating point weight values to compute 2n-bit floating point accumulated activation values, and performing activation function operations on the 2n-bit floating point accumulated activation values to compute 4n-bit floating point output values;
• performing, by the control circuitry, a conversion process to convert the 4n-bit floating point output values to n-bit floating point values, which have the first n-bit floating point format, for subsequent processing by the floating point unit circuitry, and monitoring the conversion process for an occurrence of an overflow event;
• automatically adjusting, by the control circuitry, the first exponent bias of the first n-bit floating point format to provide an adjusted exponent bias, in response to the occurrence of an overflow event during the conversion process to prevent overflow;
• configuring, by the control circuitry, the floating point unit circuitry to utilize the first n-bit floating point format with the adjusted exponent bias for a next iteration of the forward propagation neural network operations;
• and
• configuring, by the control circuitry, the floating point unit circuitry to perform backward propagation neural network operations on floating point numbers of error gradient values having a second n-bit floating point format, which comprises a same total number of bits as the first n-bit floating point format, and a second plurality of exponent bits having a number of exponent bits which differs from a number of exponent bits of the first plurality of exponent bits.

An apparatus, comprising:

• control circuitry for a neural network, the control circuitry comprising floating point unit circuitry;
• the control circuitry being configured to:
• selectively configure the floating point unit circuitry during a neural network training process to operate on floating point numbers with different n-bit floating point formats for forward propagation neural network operations and backward propagation neural network operations, the different n-bit floating point formats having a same number n of bits, but different formats with respect to exponent bits and mantissa bits, wherein in selectively configuring the floating point unit circuitry, the control circuitry is configured to:
• selectively configure the floating point unit circuitry during the neural network training process to operate using a first n-bit floating point format to perform forward propagation neural network operations on floating point numbers having the first n-bit floating point format, the first n-bit floating point format having a configuration consisting of a sign bit, m exponent bits, a first exponent bias, and p mantissa bits where m is greater than p;
• selectively configure the floating point unit circuitry during the neural network training process to operate using a second n-bit floating point format to perform backward propagation neural network operations on floating point numbers having the second n-bit floating point format, the second n-bit floating point format being different than the first n-bit floating point format, the second n-bit floating point format having a configuration consisting of a sign bit, q exponent bits, a second exponent bias, and r mantissa bits where q is greater than m and ris less than p;
• perform a conversion process to selectively convert computation results, which are generated and output from the floating point unit circuitry as a result of performing the forward and backward propagation neural network operations, into one of the first n-bit floating point format and the second n-bit floating point format for further processing by the floating point unit circuitry in a next iteration of the neural network training process;
• automatically adjust the first exponent bias of the first n-bit floating point format during the performance of the forward propagation neural network operations, in response to determining that conversion of a given computation result of a given forward propagation neural network operation will underflow or overflow the conversion of the given computation result;
• and
• automatically adjust the second exponent bias of the second n-bit floating point format during the performance of the backward propagation neural network operations, in response to determining that conversion of a given computation result of a given backward propagation neural network operation will underflow or overflow the conversion of the given computation result.