Minimum word error rate training for attention-based sequence-to-sequence models

A system comprising:

• one or more computers;
• and
• one or more computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving, by the one or more computers, audio data indicating acoustic characteristics of an utterance;
• generating, by the one or more computers, a sequence of feature vectors indicative of the acoustic characteristics of the utterance;
• processing, by the one or more computers, the sequence of feature vectors using a speech recognition model that has been trained using a loss function that uses N-best lists of decoded hypotheses, the speech recognition model comprising an encoder, an attention module providing multi-headed attention in which multiple different sets of weighting parameters are used to process different segments of output from the encoder, and a decoder, wherein the encoder and decoder each comprise one or more recurrent neural network layers and the attention module comprises a plurality of neural networks that are separately trained to generate output to the decoder from the different segments of output from the encoder;
• obtaining, by the one or more computers as a result of the processing with the speech recognition model, a sequence of output vectors representing distributions over a predetermined set of linguistic units;
• determining, by the one or more computers, a transcription for the utterance based on the sequence of output vectors;
• and
• providing, by the one or more computers, data indicating the transcription of the utterance.

One or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• receiving, by the one or more computers, audio data indicating acoustic characteristics of an utterance;
• generating, by the one or more computers, a sequence of feature vectors indicative of the acoustic characteristics of the utterance;
• processing, by the one or more computers, the sequence of feature vectors using a speech recognition model that has been trained using a loss function that uses N-best lists of decoded hypotheses, the speech recognition model comprising an encoder, an attention module providing multi-headed attention in which multiple different sets of weighting parameters are used to process different segments of output from the encoder, and a decoder, wherein the encoder and decoder each comprise one or more recurrent neural network layers and the attention module comprises a plurality of neural networks that are separately trained to generate output to the decoder from the different segments of output from the encoder;
• obtaining, by the one or more computers as a result of the processing with the speech recognition model, a sequence of output vectors representing distributions over a predetermined set of linguistic units;
• determining, by the one or more computers, a transcription for the utterance based on the sequence of output vectors;
• and
• providing, by the one or more computers, data indicating the transcription of the utterance.