Sequence processing using attention and sub-sampling

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining an input sequence of network inputs;
• processing each network input in the input sequence using a recurrent neural network to generate a sequence of recurrent outputs that includes a respective recurrent output for each network input in the input sequence;
• generating a sub-sampled sequence that includes a proper subset of the respective recurrent outputs;
• and
• processing the sub-sampled sequence using a self-attention neural network to generate a network output for the input sequence, wherein the self-attention neural network comprises:
• a self-attention subnetwork configured to apply self-attention over the sub-sampled sequence to generate a respective updated output for each recurrent output in the sub-sampled sequence;
• and
• an output neural network configured to process one or more of the updated outputs to generate the network output for the input sequence.

A system comprising one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining an input sequence of network inputs;
• processing each network input in the input sequence using a recurrent neural network to generate a sequence of recurrent outputs that includes a respective recurrent output for each network input in the input sequence;
• generating a sub-sampled sequence that includes a proper subset of the respective recurrent outputs;
• and
• processing the sub-sampled sequence using a self-attention neural network to generate a network output for the input sequence, wherein the self-attention neural network comprises:
• a self-attention subnetwork configured to apply self-attention over the sub-sampled sequence to generate a respective updated output for each recurrent output in the sub-sampled sequence;
• and
• an output neural network configured to process one or more of the updated outputs to generate the network output for the input sequence.