Sequence processing using online attention

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations to generate a target sequence comprising a respective output at each of a plurality of output time steps arranged according to an output order from respective encoded representations of inputs in an input sequence arranged according to an input order, the operations comprising:

• for each output time step after a first output time step in the output order, selecting an output at the output time step using monotonic attention over the encoded representations of the inputs, comprising:
• starting from a position, in the input order, of the encoded representation that was selected as a preceding context vector at a preceding output time step, sequentially traversing the encoded representations according to the input order until an encoded representation is selected as a current context vector at the output time step, comprising, for each encoded representation that is sequentially traversed:
• determining a selection probability for the encoded representation from the encoded representation and a hidden state of a decoder recurrent neural network for the preceding output time step;
• determining whether to select the encoded representation as the current context vector in accordance with the selection probability;
• and
• if the encoded representation is not selected as the current context vector, proceeding to a next encoded representation according to the input order;
• after sequentially traversing the encoded representations to select the current context vector at the output time step:
• processing the current context vector and a preceding output at the preceding output time step using the decoder recurrent neural network to generate a respective output score for each output in a vocabulary of outputs and to update the hidden state of the decoder recurrent neural network;
• and
• selecting the output at the output time step using the output scores;
• wherein for one or more of the output time steps, sequentially traversing the encoded representations comprises:
• sequentially traversing multiple encoded representations before an encoded representation is selected as the current context vector at the output time step,
• wherein the encoded representation that is selected as the current context vector at the output time step is separated from the encoded representation that was selected as the preceding context vector at the preceding output time step by at least one intervening encoded representation in the input sequence.

A non-transitory computer storage medium encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations to generate a target sequence comprising a respective output at each of a plurality of output time steps arranged according to an output order from respective encoded representations of inputs in an input sequence arranged according to an input order, the operations comprising:

• for each output time step after a first output time step in the output order, selecting an output at the output time step using monotonic attention over the encoded representations of the inputs, comprising:
• starting from a position, in the input order, of the encoded representation that was selected as a preceding context vector at a preceding output time step, sequentially traversing the encoded representations according to the input order until an encoded representation is selected as a current context vector at the output time step, comprising, for each encoded representation that is sequentially traversed:
• determining a selection probability for the encoded representation from the encoded representation and a hidden state of a decoder recurrent neural network for the preceding output time step;
• determining whether to select the encoded representation as the current context vector in accordance with the selection probability;
• and
• if the encoded representation is not selected as the current context vector, proceeding to a next encoded representation according to the input order;
• after sequentially traversing the encoded representations to select the current context vector at the output time step:
• processing the current context vector and a preceding output at the preceding output time step using the decoder recurrent neural network to generate a respective output score for each output in a vocabulary of outputs and to update the hidden state of the decoder recurrent neural network;
• and
• selecting the output at the output time step using the output scores;
• wherein for one or more of the output time steps, sequentially traversing the encoded representations comprises:
• sequentially traversing multiple encoded representations before an encoded representation is selected as the current context vector at the output time step,
• wherein the encoded representation that is selected as the current context vector at the output time step is separated from the encoded representation that was selected as the preceding context vector at the preceding output time step by at least one intervening encoded representation in the input sequence.