Training sequence generation neural networks using quality scores

A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, the operations comprising:

• obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;
• for each of the training examples:
• processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions, comprising, for each of the plurality of output positions, generating a likelihood distribution over the possible system outputs in the vocabulary and selecting the system output at the output position using the likelihood distribution;
• for each particular output position of the plurality of output positions in the output sequence:
• identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,
• for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence,
• generating a target likelihood distribution for the particular output position from the highest quality sores for the possible system outputs, and
• determining an update to the current values of the network parameters based on a gradient with respect to the network parameters of an objective function that depends on a divergence between the target likelihood distribution for the particular output position and the likelihood distribution generated by the neural network for the particular output position;
• and
• determining updated values of the network parameters from the updates for the particular output positions in the output sequences generated by the neural network for the batch of training examples.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations for training a neural network having a plurality of network parameters and being configured to map a system input to an output sequence comprising a plurality of system outputs, the operations comprising:

• obtaining a batch of training examples, each training example comprising a training network input and, for each training network input, a ground truth output sequence;
• for each of the training examples:
• processing the training network input in the training example using the neural network and in accordance with current values of the network parameters to map the training network input to an output sequence comprising a respective system output from a vocabulary of possible system outputs at each of a plurality of output positions, comprising, for each of the plurality of output positions, generating a likelihood distribution over the possible system outputs in the vocabulary and selecting the system output at the output position using the likelihood distribution;
• for each particular output position of the plurality of output positions in the output sequence:
• identifying a prefix that includes the system outputs at positions before the particular output position in the output sequence,
• for each possible system output in the vocabulary, determining a highest quality score that can be assigned to any candidate output sequence that includes the prefix followed by the possible system output and followed by any suffix of one or more system outputs, wherein the quality score measures a quality of a candidate output sequence relative to the ground truth output sequence,
• generating a target likelihood distribution for the particular output position from the highest quality sores for the possible system outputs, and
• determining an update to the current values of the network parameters based on a gradient with respect to the network parameters of an objective function that depends on a divergence between the target likelihood distribution for the particular output position and the likelihood distribution generated by the neural network for the particular output position;
• and
• determining updated values of the network parameters from the updates for the particular output positions in the output sequences generated by the neural network for the batch of training examples.