Enhanced attention mechanisms

A system comprising:

• one or more computers;
• and
• one or more computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• receiving, by the one or more computers, data indicating an input sequence;
• processing, by the one or more computers, the data using an encoder neural network to generate a sequence of encodings;
• determining, by the one or more computers, a series of attention outputs with respect to the encodings using one or more attender modules,
• wherein determining each of the attention outputs comprises (i) selecting an encoding from the sequence of encodings and (ii) determining attention over a proper subset of the sequence of encodings, wherein the proper subset of encodings is determined based on a position of the selected encoding in the sequence of encodings, and
• wherein the selections of encodings are monotonic through the sequence of encodings;
• generating, by the one or more computers, an output sequence by processing the attention outputs using a decoder neural network;
• and
• providing, by the one or more computers, an output that indicates a language sequence determined from the output sequence.

One or more non-transitory computer-readable media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• receiving, by the one or more computers, data indicating an input sequence;
• processing, by the one or more computers, the data using an encoder neural network to generate a sequence of encodings;
• determining, by the one or more computers, a series of attention outputs with respect to the encodings using one or more attender modules,
• wherein determining each of the attention outputs comprises (i) selecting an encoding from the sequence of encodings and (ii) determining attention over a proper subset of the sequence of encodings, wherein the proper subset of encodings is determined based on a position of the selected encoding in the sequence of encodings, and
• wherein the selections of encodings are monotonic through the sequence of encodings;
• generating, by the one or more computers, an output sequence by processing the attention outputs using a decoder neural network;
• and
• providing, by the one or more computers, an output that indicates a language sequence determined from the output sequence.