Number Format Selection in Recurrent Neural Networks

A data processing system for selecting one or more number formats for representing two or more values of a recurrent neural network (RNN) for use in configuring a hardware implementation of the RNN, the data processing system comprising:

• a processor;
• control logic configured to implement a representation of the RNN at the processor as a test neural network for operation on a sequence of test inputs, each step of the test neural network comprising an instance of the two or more values of the RNN;
• and
• a format selection unit configured to cause the processor to operate the test neural network for a plurality of steps on the sequence of test inputs and collect statistics for provision to a number format selection algorithm;
• wherein the format selection unit is configured to apply the number format selection algorithm to the statistics so as to derive a common number format for the plurality of instances of the two or more values of the RNN.

A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform a computer-implemented method of selecting a number format for representing two or more values of a recurrent neural network (RNN) for use in configuring a hardware implementation of the RNN, the method comprising:

• receiving a representation of the RNN;
• implementing the representation of the RNN as a test neural network for operation on a sequence of test inputs, each step of the test neural network comprising an instance of the two or more values of the RNN;
• operating the test neural network for a plurality of steps on the sequence of test inputs and collecting statistics for provision to a number format selection algorithm;
• and
• applying a number format selection algorithm to the statistics so as to derive a common number format for the plurality of instances of the two or more values of the RNN.