Autoregressive language models for video generation

#Categories of video generation methods include generative adversarial network (GAN)-based, diffusion-based, and language-model-based methodologies. Among them, diffusion-based methods have recently attracted great attention. Most diffusion-based methods encode videos into latent space for efficient training and utilize progressive inference strategies to generate videos with high spatial-temporal resolution. Language models have recently been explored for visual generation, focusing on tokenizing visual data into a form that can be processed by these models. Quantization techniques are commonly used, and transformers are employed to model the resulting tokens.

#For image generation, autoregressive or masked transformers are prevalent. In short video generation, image-level or video-level tokenizers are utilized, incorporating spatial-temporal compression and causal structures. However, short video generation models generally focus on producing clips (e.g., videos with durations under 10 seconds that are typically of 1-3 seconds in length), limiting their ability to capture complex events and maintain consistency over longer durations. Previous works have explored long video generation (e.g., videos with durations longer than ten seconds) using various approaches. More recently, video diffusion models have been extended for long video generation. Some methods focus on sampling noise vectors and aggregating overlapping short video segments, respectively. Other methods propose an autoregressive approach with memory blocks for consistency and appearance preservation. In the language model domain, methods capable of generating variable-length videos using a masked video transformer have been contemplated. However, despite these advancements, generating long videos with rich motion dynamics, consistent appearance, and high visual quality in the open domain remains a challenge.

#Implementations for autoregressively generating a video using a video generation model are provided. One aspect includes a method comprising: performing a progressive multi-stage training process comprising a first stage and a second stage, wherein: the first stage comprises training the video generation model to perform text-to-image generation; and the second stage comprises further training the video generation model using a training dataset comprising labeled video-text pairs, wherein further training the video generation model comprises: for each of the labeled video-text pair: generating at least one text token using a text tokenizer and a text annotation of the labeled video-text pair; generating a plurality of video tokens using a video tokenizer and a video of the labeled video-text pair; autoregressively generating frame tokens using the at least one text token; and training the video generation model using loss values calculated from the frame tokens and the video tokens.

#FIG. 1 shows a schematic view of an example computing system for training an autoregressive video generation model.

#FIG. 2 shows a data flow diagram of an example training process of an autoregressive video generator, which can be implemented using the computer system of FIG. 1.

#FIG. 3 shows a graph depicting training loss for different frame ranges.

#Video generation models, including diffusion-based and language model-based approaches, are highly capable of generating short videos (e.g., videos with durations under 10 seconds that are typically of 1-3 seconds in length). To capture more comprehensive content, it is desirable to generate long videos (e.g., videos with durations longer than 10 seconds and can include minute-level videos) with consistent appearance, larger motion dynamics, and natural scene transitions. Autoregressive generative language models have shown success in generating long and coherent text sequences, demonstrating their ability to capture long-range dependencies and complex temporal patterns. However, the use of autoregressive generative language models for video generation is limited to generating short videos of several seconds.

#In natural language processing, generative language models can be trained on long sequences and extended beyond the training length. However, training autoregressive generative language models on long video sequences or extending short video generators to generate long videos would lead to unsatisfactory performance for minute-level video generation. The main obstacles are the large redundancy and strong inter-frame dependency among video tokens. The video tokens of the current frame depend heavily on the tokens of the previous frames, leading to two challenges for long video generation. The first challenge is the imbalance of loss during training. When trained with the next-token prediction objective, predicting early-frame tokens from text prompts is much more difficult than predicting late-frame tokens based on the ground-truth tokens of previous frames. The imbalanced difficulty of tokens of different stages lead to imbalanced loss during training. The issue becomes more severe as the video length increases, where the accumulated loss of many easy tokens largely surpasses the loss of a few difficult tokens and dominates the gradient direction. The second challenge includes error accumulation during inference. While the model predicts the next token conditioned on previous ground-truth tokens during training, it has to predict the next token conditioned on previous predicted tokens during inference. This training-inference discrepancy leads to error accumulation during inference. Because of the strong inter-frame dependency among video tokens and the large number of video tokens, such error accumulation is non-negligible and can cause visual quality degradation for long video inference.

#In view of the observations above, implementations of autoregressive generative language models for long video generation are provided. As described herein, novel implementations of an autoregressive generative language model based video generator can be employed to generate content-rich, coherent, and dynamic long videos in the scale of minutes. In some implementations, the autoregressive generative language model based video generator is trained on ten-second videos and can, as such, generate ten-second videos. This capability can be extended to generate minute-level long videos conditioned on text prompts.