Self-attention based neural networks for processing network inputs from multiple modalities

A system comprising a first neural network that is configured to process input sequences corresponding to a single particular modality of a plurality of modalities and to generate respective outputs for the input sequences,

• the first neural network having been trained by performing operations comprising:
• training a second neural network to process respective input sequences corresponding to each of the plurality of modalities, the training comprising:
• obtaining a network input corresponding to a particular modality and comprising a plurality of elements;
• determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input;
• processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches;
• processing the input sequence using the second neural network to generate, for at least one of one or more machine learning tasks corresponding to the particular modality, a respective predicted network output,
• wherein the second neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence, and
• wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities;
• and
• determining an update to a plurality of parameters of the second neural network according to an error in the respective predicted network outputs;
• and
• modifying an architecture of the second neural network to generate the first neural network.

A method performed by one or more computers, the method comprising:

• receiving a respective network input corresponding to each of a plurality of modalities, wherein each network input includes a respective plurality of elements;
• and
• processing the respective network inputs using a neural network to generate respective network outputs for each of the network inputs,
• the processing comprising, for each network input:
• determining a plurality of patches of the network input, wherein each patch comprises a different subset of the elements of the network input;
• processing the plurality of patches to generate an input sequence comprising a respective input element at each of a plurality of input positions, where some or all of the input elements correspond to respective different patches;
• and
• processing the input sequence using the neural network to generate, for at least one of one or more machine learning tasks corresponding to the modality corresponding to the network input, a respective predicted network output,
• wherein the neural network comprises one or more self-attention neural network layers that are each configured to apply a self-attention mechanism to the input sequence or an intermediate representation of the input sequence;
• and
• wherein at least a subset of the self-attention neural network layers are shared across the plurality of modalities.