Variable length video generation from textual descriptions

A system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising:

• receiving a first text prompt;
• generating, using a video generation neural network and conditioned on the first text prompt, an initial segment of the video that comprises a respective video frame at each of a plurality of initial time steps in the video, wherein the video generation neural network comprises a token prediction neural network and a video decoder neural network, the generating comprising:
• processing the first text prompt using a text embedding neural network to generate an encoded representation of the first text prompt;
• generating, using the token prediction neural network, a sequence of video tokens that represent the initial segment conditioned on the encoded representation of the first text prompt;
• and
• processing the sequence of video tokens using the video decoder neural network to generate the video frames in the initial segment;
• and
• for each of one or more update iterations, updating the video by generating an additional video segment for the update iteration that includes a respective video frame at each of a plurality of time steps immediately following a last video frame in the video as of the update iteration, the updating comprising:
• obtaining an additional text prompt for the update iteration;
• and
• generating, using the video generation neural network and conditioned on the additional text prompt for the update iteration and on one or more video frames in the video as of the update iteration, the additional video segment for the update iteration.

One or more non-transitory computer-readable storage media encoded with instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• receiving a first text prompt;
• generating, using a video generation neural network and conditioned on the first text prompt, an initial segment of the video that comprises a respective video frame at each of a plurality of initial time steps in the video, wherein the video generation neural network comprises a token prediction neural network and a video decoder neural network, the generating comprising:
• processing the first text prompt using a text embedding neural network to generate an encoded representation of the first text prompt;
• generating, using the token prediction neural network, a sequence of video tokens that represent the initial segment conditioned on the encoded representation of the first text prompt;
• and
• processing the sequence of video tokens using the video decoder neural network to generate the video frames in the initial segment;
• and
• for each of one or more update iterations, updating the video by generating an additional video segment for the update iteration that includes a respective video frame at each of a plurality of time steps immediately following a last video frame in the video as of the update iteration, the updating comprising:
• obtaining an additional text prompt for the update iteration;
• and
• generating, using the video generation neural network and conditioned on the additional text prompt for the update iteration and on one or more video frames in the video as of the update iteration, the additional video segment for the update iteration.