Methods and systems for running dynamic recurrent neural networks in hardware

A data processing system for implementing a recurrent neural network (RNN) for operation on a sequence of inputs, the system comprising:

• a transformation unit configured to receive a representation of the RNN and transform the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN;
• a hardware accelerator for processing neural networks;
• and
• iteration logic configured to iteratively apply the derivative neural network to the sequence of inputs by:
• causing a sequence of instances of the derivative neural network to be implemented at the hardware accelerator;
• and
• providing the one or more state outputs from each representation of the derivative neural network at the hardware accelerator as the one or more state inputs to a subsequent representation of the derivative neural network at the hardware accelerator so as to cause the hardware accelerator to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.

A non-transitory computer readable storage medium having stored thereon computer readable instructions that, when executed at a computer system, cause the computer system to perform a method of implementing in hardware a recurrent neural network (RNN) for operation on a sequence of inputs, each step of the recurrent neural network being for operation on a different input of the sequence, the method comprising:

• receiving a representation of the RNN;
• transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN;
• and
• iteratively applying the derivative neural network to the sequence of inputs by:
• implementing a sequence of instances of the derivative neural network in hardware;
• and
• providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.