Training recurrent neural networks to generate sequences

A system for training a recurrent neural network to generate target sequences for received inputs, the system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations,

• wherein the recurrent neural network is configured to, at each position that is after an initial position in an output order of the target sequence, receive as input a preceding output at a preceding position in the output order and generate a set of output scores for the position,
• wherein the set of output scores comprises a respective score for each of a plurality of possible outputs, and
• wherein the operations comprise:
• obtaining a plurality of training data pairs for the recurrent neural network, each training data pair comprising a training input and a training target sequence for the training input, each training target sequence comprising a respective plurality of true outputs arranged according to an output order;
• and
• training the recurrent neural network on the training data pairs, comprising, for each training data pair:
• for each position in the training target sequence in the training data pair that is after an initial position in the output order:
• selecting a preceding output to provide as input to the recurrent neural network at the position, wherein the selected preceding output is selected from a group consisting of (i) the true output for a preceding position in the output order and (ii) a value derived from an output of the recurrent neural network for the preceding position in the output order generated in accordance with current values of the parameters of the recurrent neural network.

One or more computer storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations for training a recurrent neural network to generate target sequences for received inputs,

• wherein the recurrent neural network is configured to, at each position that is after an initial position in an output order of the target sequence, receive as input a preceding output at a preceding position in the output order and generate a set of output scores for the position,
• wherein the set of output scores comprises a respective score for each of a plurality of possible outputs, and
• wherein the operations comprise:
• obtaining a plurality of training data pairs for the recurrent neural network, each training data pair comprising a training input and a training target sequence for the training input, each training target sequence comprising a respective plurality of true outputs arranged according to an output order;
• and
• training the recurrent neural network on the training data pairs, comprising, for each training data pair:
• for each position in the training target sequence in the training data pair that is after an initial position in the output order:
• selecting a preceding output to provide as input to the recurrent neural network at the position, wherein the selected preceding output is selected from a group consisting of (i) the true output for a preceding position in the output order and (ii) a value derived from an output of the recurrent neural network for the preceding position in the output order generated in accordance with current values of the parameters of the recurrent neural network.