Neural machine translation systems

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to implement a neural machine translation system that is configured to receive an input sequence of input tokens representing a first sequence of words in a first natural language and to generate an output sequence of output tokens representing a second sequence of words that is a translation of the first sequence into a second natural language, the neural machine translation system comprising:

• an encoder neural network comprising:
• an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token,
• an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token,
• a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and
• a plurality of hidden LSTM layers configured to process each combined representation generated by the combining layer in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are unidirectional and arranged in a stack one after the other;
• and
• a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.

A method performed by one or more computers, the method comprising:

• receiving an input sequence of input tokens representing a first sequence of words in a first natural language, and
• processing the input sequence using a neural machine translation system to generate an output sequence of output tokens representing a second sequence of words that is a translation of the first sequence into a second natural language, the neural machine translation system comprising:
• an encoder neural network comprising:
• an input forward long short-term memory (LSTM) layer configured to process each input token in the input sequence in a forward order to generate a respective forward representation of each input token,
• an input backward LSTM layer configured to process each input token in the input sequence in a backward order to generate a respective backward representation of each input token,
• a combining layer configured to, for each input token in the input sequence, combine the forward representation of the input token and the backward representation of the input token to generate a combined representation of the input token, and
• a plurality of hidden LSTM layers configured to process each combined representation generated by the combining layer in the forward order to generate a respective encoded representation of each of the input tokens in the input sequence, wherein the plurality of hidden LSTM layers are unidirectional and arranged in a stack one after the other;
• and
• a decoder subsystem configured to receive the respective encoded representation of each of the input tokens in the input sequence and to process the encoded representations to generate the output sequence.