Encoder-decoder models for sequence to sequence mapping

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• obtaining, by the one or more computers, acoustic data representing an utterance, the acoustic data corresponding to time steps in a series of time steps;
• processing, by the one or more computers, scores indicative of the acoustic data using a recurrent neural network to generate a sequence of outputs, wherein the sequence of outputs indicates likely output labels from among a predetermined set of output labels, wherein the predetermined set of output labels includes output labels that respectively correspond to different linguistic units and to a placeholder label that does not represent a classification of acoustic data, wherein the recurrent neural network is configured to use an output label that was selected, based on output of the recurrent neural network, for a previous time step to determine output of the recurrent neural network indicating likelihoods of the output labels for the current time step;
• processing the generated sequence of outputs to generate a transcription of the utterance;
• and
• providing, by the one or more computers, the generated transcription of the utterance as output of the speech recognition system,
• wherein the recurrent neural network has been trained to process received input acoustic sequences and generate sequences of outputs, the training comprising:
• obtaining a plurality of training examples, each training example comprising (i) an input acoustic sequence of scores indicative of the acoustic data at each of multiple time steps in a series of time steps, the input acoustic sequence representing a known utterance, and (ii) a corresponding target sequence of linguistic units representing a transcription of the utterance;
• training the recurrent neural network to minimize a negative log likelihood loss function by, for each of the plurality of training samples:
• representing possible alignments between the input acoustic sequence and the target sequence of linguistic units as a lattice, the possible alignments constrained to allow placeholder label repetitions only and each node in the lattice represents a respective state of the recurrent neural network, each state of the recurrent neural network being dependent on a respective time step from the series of time steps and a respective position in the target sequence of linguistic units, and wherein transitions between nodes in the lattice represent probabilities of observing respective subsequent linguistic units or placeholder labels in the target sequence of linguistic units;
• performing forward calculations through the lattice to update each recurrent neural network state;
• approximating the log likelihood loss function using the updated recurrent neural network states;
• and
• performing back propagation techniques using the approximated log likelihood function to adjust recurrent neural network parameters to trained recurrent neural network parameters;
• and
• training the recurrent neural network to minimize an expected loss function using the plurality of training examples.

A non-transitory computer-readable storage medium comprising instructions stored thereon that are executable by a processing device and upon such execution cause the processing device to perform operations comprising:

• obtaining, by the one or more computers, acoustic data representing an utterance, the acoustic data corresponding to time steps in a series of time steps;
• processing, by the one or more computers, scores indicative of the acoustic data using a recurrent neural network to generate a sequence of outputs, wherein the sequence of outputs indicates likely output labels from among a predetermined set of output labels, wherein the predetermined set of output labels includes output labels that respectively correspond to different linguistic units and to a placeholder label that does not represent a classification of acoustic data, wherein the recurrent neural network is configured to use an output label that was selected, based on output of the recurrent neural network, for a previous time step to determine output of the recurrent neural network indicating likelihoods of the output labels for the current time step;
• processing the generated sequence of outputs to generate a transcription of the utterance;
• and
• providing, by the one or more computers, the generated transcription of the utterance as output of the speech recognition system,
• wherein the recurrent neural network has been trained to process received input acoustic sequences and generate sequences of outputs, the training comprising:
• obtaining a plurality of training examples, each training example comprising (i) an input acoustic sequence of scores indicative of the acoustic data at each of multiple time steps in a series of time steps, the input acoustic sequence representing a known utterance, and (ii) a corresponding target sequence of linguistic units representing a transcription of the utterance;
• training the recurrent neural network to minimize a negative log likelihood loss function by, for each of the plurality of training samples:
• representing possible alignments between the input acoustic sequence and the target sequence of linguistic units as a lattice, the possible alignments constrained to allow placeholder label repetitions only and each node in the lattice represents a respective state of the recurrent neural network, each state of the recurrent neural network being dependent on a respective time step from the series of time steps and a respective position in the target sequence of linguistic units, and wherein transitions between nodes in the lattice represent probabilities of observing respective subsequent linguistic units or placeholder labels in the target sequence of linguistic units;
• performing forward calculations through the lattice to update each recurrent neural network state;
• approximating the log likelihood loss function using the updated recurrent neural network states;
• and
• performing back propagation techniques using the approximated log likelihood function to adjust recurrent neural network parameters to trained recurrent neural network parameters;
• and
• training the recurrent neural network to minimize an expected loss function using the plurality of training examples.