Artificial neural network training using flexible floating point tensors

A method of training a neural network, comprising:

• generating, by processor circuitry, a neural network training tensor that includes:
• a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the 16-bit floating point values including:
• a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
• and
• a second plurality of bits stored in the memory, the second plurality of bits form an exponent portion of the respective floating point value;
• and
• a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;
• adjusting, by the processor circuitry in response to prediction of an overflow condition or an underflow condition during training of the neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow;
• and
• providing, by the processor circuitry, at least one of the plurality of 16-bit floating point values included in the neural network training tensor as an input to the neural network.

A neural network training system, comprising:

• means for generating a neural network training tensor that includes:
• a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the 16-bit floating point values including:
• a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
• a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value;
• and
• a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;
• means for adjusting, in response to prediction of an overflow condition or an underflow condition during training of a neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow;
• and
• means for providing as an input to the neural network at least one of the plurality of 16-bit floating point values included in the neural network training tensor.

A non-transitory, computer-readable storage medium that includes machine readable instructions that, when executed by processor circuitry, cause the processor circuitry to:

• generate a neural network training tensor that includes:
• a five-bit shared exponent stored in a memory, the five-bit shared exponent common to each of a plurality of 16-bit floating point values included in the neural network training tensor, each of the plurality of 16-bit floating point values including:
• a first plurality of bits stored in the memory, the first plurality of bits to form a mantissa of the respective floating point value;
• a second plurality of bits stored in the memory, the second plurality of bits to form an exponent portion of the respective floating point value;
• and
• a one-bit switch stored in the memory, the one-bit switch to selectively combine the exponent portion of the respective 16-bit floating point value with the five-bit shared exponent;
• and
• adjust, in response to a prediction of an overflow or underflow condition during training of a neural network, one or more of the 16-bit floating point values in the memory to avoid the predicted overflow or underflow.

Processor circuitry programmed to implement a tensor data storage structure in a plurality of hardware registers, comprising:

• a five-bit shared hardware register to store a first exponent value, the five-bit shared hardware register associated with each respective one of a plurality of 16-bit floating point hardware registers, each of the plurality of 16-bit floating point hardware registers including:
• a first hardware register to store a mantissa value;
• a second hardware register to store a second exponent value;
• a third hardware register to store a sign bit;
• and
• a one-bit switch hardware register to selectively combine each respective 16-bit floating point hardware register with the five-bit shared exponent hardware register;
• and
• predictive circuitry to adjust one or more of the 16-bit floating point hardware registers in response to a prediction of an overflow or underflow condition in one or more of the 16-bit floating point hardware registers to avoid the overflow or underflow.