Generating multimodal data using a media item generation neural network and a token generation neural network

A system comprising:

• one or more computers;
• and
• one or more storage devices communicatively coupled to the one or more computers, wherein the one or more storage devices store instructions that, when executed by the one or more computers, cause the one or more computers to generate an output media item using a token generation neural network and a media generation neural network, the operations comprising:
• obtaining a multimodal input that comprises a set of one or more media items and one or more prompt tokens of a different modality;
• processing each of the one or more media items using a first media encoder neural network to generate a respective set of media tokens for each of the one or more media items;
• generating an input sequence of multimodal tokens that comprises the prompt tokens and the media tokens for the one or more media items;
• processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, comprising:
• determining that a criterion is satisfied as of a given multimodal token in the output sequence;
• and
• in response:
• processing each of the one or more media items using a second media encoder neural network to generate a respective encoded representation for each of the one or more media items;
• and
• generating an output media item using the media generation neural network conditioned on (i) features representing a current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network and (ii) the encoded representations of the one or more media items.

One or more non-transitory computer storage media storing instructions that when executed by one or more computers cause the one or more computers to generate an output media item using a token generation neural network and a media generation neural network, the operations comprising:

• obtaining a multimodal input that comprises a set of one or more media items and one or more prompt tokens of a different modality;
• processing each of the one or more media items using a first media encoder neural network to generate a respective set of media tokens for each of the one or more media items;
• generating an input sequence of multimodal tokens that comprises the prompt tokens and the media tokens for the one or more media items;
• processing the input sequence of multimodal tokens using the token generation neural network to generate an output sequence of multimodal tokens, comprising:
• determining that a criterion is satisfied as of a given multimodal token in the output sequence;
• and
• in response:
• processing each of the one or more media items using a second media encoder neural network to generate a respective encoded representation for each of the one or more media items;
• and
• generating an output media item using the media generation neural network conditioned on (i) features representing a current output sequence of multimodal tokens as of the given multimodal token obtained from the token generation neural network and (ii) the encoded representations of the one or more media items.