Multimodal few-shot learning with frozen language models

A system comprising:

• one or more computers;
• and
• one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:
• obtaining an input image;
• processing the input image using an image encoder neural network that has been trained to process the input image to generate an image sequence of image embedding vectors characterizing the input image;
• generating an input sequence of input embedding vectors that includes the image sequence of image embedding vectors;
• and
• processing the input sequence of input embedding vectors that includes the image sequence of image embedding vectors using a trained language model neural network to generate as output an output sequence of text tokens that represents a text response to the input sequence of input embedding vectors, wherein:
• each text token in the output sequence is selected from a vocabulary of text tokens,
• the trained language model neural network has been trained to perform a language modeling task on training input sequences of text embedding vectors that represent text tokens from the vocabulary of text tokens, and
• each image embedding vector in the image sequence of image embedding vectors has a same dimensionality as the text embedding vectors that represent the text tokens from the vocabulary of text tokens,
• the language model neural network is an auto-regressive Transformer-based neural network that comprises a plurality of self-attention layers and that auto-regressively generates the output sequence of text tokens by generating each particular text token in the output sequence conditioned on a current input sequence that includes (i) the input sequence of input embedding vectors followed by text embedding vectors of any text tokens that precede the particular text token in the output sequence, and
• wherein the language model neural network is configured to apply relative position encodings to the input sequence when processing the input sequence using the plurality of layers.

One or more non-transitory computer-readable storage media storing instructions that when executed by one or more computers cause the one or more computers to perform operations comprising:

• obtaining an input image;
• processing the input image using an image encoder neural network that has been trained to process the input image to generate an image sequence of image embedding vectors characterizing the input image;
• generating an input sequence of input embedding vectors that includes the image sequence of image embedding vectors;
• and
• processing the input sequence of input embedding vectors that includes the image sequence of image embedding vectors using a trained language model neural network to generate as output an output sequence of text tokens that represents a text response to the input sequence of input embedding vectors, wherein:
• each text token in the output sequence is selected from a vocabulary of text tokens,
• the trained language model neural network has been trained to perform a language modeling task on training input sequences of text embedding vectors that represent text tokens from the vocabulary of text tokens, and
• each image embedding vector in the image sequence of image embedding vectors has a same dimensionality as the text embedding vectors that represent the text tokens from the vocabulary of text tokens,
• the language model neural network is an auto-regressive Transformer-based neural network that comprises a plurality of self-attention layers and that auto-regressively generates the output sequence of text tokens by generating each particular text token in the output sequence conditioned on a current input sequence that includes (i) the input sequence of input embedding vectors followed by text embedding vectors of any text tokens that precede the particular text token in the output sequence, and
• wherein the language model neural network is configured to apply relative position encodings to the input sequence when processing the input sequence using the plurality of layers.