Processing large-scale textual inputs using neural networks

A computer-implemented method comprising:

• receiving, by a neural network, a tuple of a plurality of input sequences, wherein the neural network comprises (i) a plurality of encoder neural networks that comprise a different encoder neural network corresponding to each different input sequence included in the tuple and (ii) a head neural network, wherein each encoder neural network comprises one or more encoder network layers and a projection layer;
• receiving, at each encoder neural network, an input sequence that corresponds to the encoder neural network from the tuple, wherein the input sequence has a respective network input at each of a plurality of input positions in an input order;
• processing, using the one or more encoder network layers of each encoder neural network, the input sequence to generate an encoded representation of the input sequence, the encoded representation comprising a sequence of tokens;
• determining a respective proper subset of the sequence of tokens in the encoded representation based on respective positions of the tokens in the sequence and a length of the input sequence that corresponds to the encoder neural network;
• processing, using the projection layer of each encoder neural network, each token in the respective proper subset to generate a lower-dimensional representation of the token;
• receiving, at the head neural network and from each of the plurality of encoder neural networks, the lower-dimensional representation of each token in the respective proper subset that is generated by the encoder neural network;
• and
• processing, using the head neural network, the lower-dimensional representations to generate an output.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to implement:

• a neural network configured to perform a machine learning task on a tuple of a plurality of input sequences to generate a network output, the neural network comprising (i) a plurality of encoder neural networks that comprise a different encoder neural network corresponding to each different input sequence included in the tuple and (ii) a head neural network, wherein each encoder neural network comprises one or more encoder network layers and a projection layer, wherein each encoder neural network is configured to:
• receive an input sequence that corresponds to the encoder neural network from the tuple, wherein the input sequence has a respective network input at each of a plurality of input positions in an input order;
• process the input sequence using the one or more encoder network layers to generate an encoded representation of the respective input sequence, the encoded representation comprising a sequence of tokens;
• determine a respective proper subset of the sequence of tokens in the encoded representation based on respective positions of the tokens in the sequence and a length of the input sequence that corresponds to the encoder neural network;
• and
• process each token in the respective proper subset using the projection layer to generate a lower-dimensional representation of the token, and wherein the head neural network is configured to:
• receive, from each of the plurality of encoder neural networks, the lower-dimensional representation of each token in the respective proper subset that is generated by the encoder neural network;
• and
• process the lower-dimensional representations to generate the output.