Auto-regressive video generation neural networks

A computer-implemented method comprising:

• identifying a partitioning of an initial output video into a set of segments that are indexed according to a particular order;
• repeatedly performing the following operations for each segment in the set according to the particular order:
• processing, using an encoder neural network, a current output video comprising a current representation of the set of segments to generate an encoded conditioning segment, wherein the encoder neural network comprises a plurality of encoding self-attention layers, wherein each of the plurality of encoding self-attention layers is configured to receive as input a padded video that is a representation of the current output video, divide the padded video into a plurality of blocks, and apply a self-attention mechanism on each of the plurality of blocks, and
• processing, using a decoder neural network, the encoded conditioning segment to generate respective values for the segment;
• and
• generating a final output video using the set of segments with each segment having the respective generated values.

A system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations comprising:

• identifying a partitioning of an initial output video into a set of segments that are indexed according to a particular order;
• repeatedly performing the following operations for each segment in the set according to the particular order:
• processing, using an encoder neural network, a current output video comprising a current representation of the set of segments to generate an encoded conditioning segment, wherein the encoder neural network comprises a plurality of encoding self-attention layers, wherein each of the plurality of encoding self-attention layers is configured to receive as input a padded video that is a representation of the current output video, divide the padded video into a plurality of blocks, and apply a self-attention mechanism on each of the plurality of blocks, and
• processing, using a decoder neural network, the encoded conditioning segment to generate respective values for the segment;
• and
• generating a final output video using the set of segments with each segment having the respective generated values.

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations comprising:

• identifying a partitioning of an initial output video into a set of segments that are indexed according to a particular order;
• repeatedly performing the following operations for each segment in the set according to the particular order:
• processing, using an encoder neural network, a current output video comprising a current representation of the set of segments to generate an encoded conditioning segment, wherein the encoder neural network comprises a plurality of encoding self-attention layers, wherein each of the plurality of encoding self-attention layers is configured to receive as input a padded video that is a representation of the current output video, divide the padded video into a plurality of blocks, and apply a self-attention mechanism on each of the plurality of blocks, and
• processing, using a decoder neural network, the encoded conditioning segment to generate respective values for the segment;
• and
• generating a final output video using the set of segments with each segment having the respective generated values.