Audio-visual representation learning for lip-sync estimation through ranking augmented contrastive training

One or more non-transitory computer readable media storing instructions that, when executed by one or more processors, cause the one or more processors train lip-sync estimation models, by performing the operations of:

• obtaining video training data comprising a plurality of training videos and corresponding audio training data;
• selecting an anchor video from the plurality of training videos;
• identifying, with respect to the anchor video and based on a similarity evaluation generated by a machine learning (ML) model, a plurality of audio samples;
• generating a training loss from the plurality of audio samples;
• applying backpropagation from the training loss to update parameters of the ML model until a convergence criteria is satisfied;
• and
• generating a trained lip-sync estimation model based on the updated parameters.

A computer system, comprising:

• one or more memories that include instructions;
• and
• one or more processors that are coupled to the one or more memories and, when executing the instructions, are configured to train lip-sync estimation models, by performing the operations of:
• obtaining video training data comprising a plurality of training videos and corresponding audio training data;
• selecting an anchor video from the plurality of training videos;
• identifying, with respect to the anchor video and based on a similarity evaluation generated by a machine learning (ML) model, a plurality of audio samples;
• generating a training loss from the plurality of audio samples;
• applying backpropagation from the training loss to update parameters of the ML model until a convergence criteria is satisfied;
• and
• generating a trained lip-sync estimation model based on the updated parameters.