Audio-visual representation learning for lip-sync estimation through ranking augmented contrastive training

One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform multi-stage training of lip-sync estimation models, by performing the operations of:

• obtaining video training data comprising a plurality of training videos and corresponding audio training data;
• training a machine learning (ML) model for lip-sync estimation through a plurality of training stages, wherein each successive stage utilizes training data having greater synchronization complexity than a preceding training stage;
• updating parameters of the ML model based on results generated from the plurality of training stages;
• and
• generating a trained lip-sync estimation model based on the updated parameters of the ML model.

A computer system, comprising:

• one or more memories that include instructions;
• and
• one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to perform multi-stage training of lip-sync estimation models, by performing the operations of:
• obtaining video training data comprising a plurality of training videos and corresponding audio training data;
• training a machine learning (ML) model for lip-sync estimation through a plurality of training stages, wherein each successive stage utilizes training data having greater synchronization complexity than a preceding training stage;
• updating parameters of the ML model based on results generated from the plurality of training stages;
• and
• generating a trained lip-sync estimation model based on the updated parameters of the ML model.