Music-driven dance video generation

A method for music-driven dance video generation, the method comprising:

• receiving a reference image and a music sequence;
• generating music embeddings using the music sequence;
• generating a sequence of pose tokens using one or more two-dimensional (2D) pose encoders and the reference image;
• autoregressively generating one or more successive sequences of pose tokens using a transformer model, the sequence of pose tokens, and the music embeddings;
• generating one or more feature maps using a motion guidance decoder and the one or more successive sequence of pose tokens;
• and
• generating one or more successive image frames using a diffusion model conditioned on the reference image and the one or more feature maps.

A method for training a framework for music-driven dance video generation, the method comprising:

• receiving a training dataset comprising a plurality of monocular videos;
• training a plurality of two-dimensional (2D) pose encoders using the training dataset, wherein each of the plurality of 2D pose encoders is trained to encode a different human body part;
• training a transformer model to autoregressively perform next token prediction using sequences of pose tokens generated by the trained plurality of 2D pose encoders and the training dataset;
• and
• training a diffusion model to generate images using the training dataset.