Real-time interactive video generation method and system

A real-time interactive video generation method comprising:

• acquiring training data comprising interactive information and video data corresponding to the interactive information;
• preprocessing the training data to obtain target training data;
• performing model training comprising a model pre-training process and a video pre-generation process based on the target training data, wherein the model pre-training process comprises a training of a low-resolution image reconstruction model and a training of a super-resolution model, and the video pre-generation process comprises performing a pre-generation of a video based on an interactive video generation model to complete a training of the interactive video generation model, the trained interactive video generation model having the low-resolution image reconstruction model and the super-resolution model, the trainings of which have been completed;
• and
• generating an interactive video by using the trained interactive video generation model,
• wherein the video pre-generation process further comprises:
• generating an image latent feature corresponding to an initial frame of the low-resolution images;
• inputting the image latent feature and interactive information of the corresponding low-resolution image into a transformer of the interactive video generation model to generate a target latent feature corresponding to a first frame image;
• inputting the target latent feature into an image reconstruction decoder, and performing image reconstruction on the target latent feature through the image reconstruction decoder to obtain a reconstructed low-resolution image, wherein the image reconstruction decoder is a decoder of the low-resolution image reconstruction model, the training of which is completed;
• and
• inputting the reconstructed low-resolution image into the super-resolution model, the training of which has been completed, and performing the super-resolution processing on the reconstructed low-resolution image through the super-resolution model to obtain the each of the frame images corresponding to the pre-generated video with the same resolution as the video data,
• wherein low-resolution refers to a resolution that is lower than that of the obtained frame images with the same resolution as the video data.