Machine learning-based approach for audio-driven avatar animation or other functions

An electronic device comprising:

• at least one processing device configured to:
• obtain an audio input associated with a speaker;
• extract audio features from the audio input using a feature extractor of a trained machine learning model;
• and
• generate (i) one or more content parameter predictions using content embeddings extracted by a content encoder and decoded by a content decoder of the trained machine learning model and (ii) one or more style parameter predictions using style embeddings extracted by a style encoder and decoded by a style decoder of the trained machine learning model, the content embeddings and the style embeddings based on the audio features of the audio input;
• wherein the trained machine learning model is trained to generate the one or more content parameter predictions and the one or more style parameter predictions using disentangled content and style embeddings.

A method comprising:

• obtaining, using at least one processing device of an electronic device, audio samples of at least one training dataset;
• and
• training, using the at least one processing device, a machine learning model to generate content parameter predictions and style parameter predictions using the audio samples;
• wherein training the machine learning model comprises:
• generating audio features of the audio samples using a feature extractor of the machine learning model;
• providing a first collection of the audio features as inputs to a content encoder of the machine learning model in order to generate first content embeddings;
• providing the first content embeddings as inputs to a style decoder of the machine learning model in order to generate first style parameter predictions;
• providing the first collection of the audio features as inputs to a style encoder of the machine learning model in order to generate first style embeddings;
• providing the first style embeddings as inputs to a content decoder of the machine learning model in order to generate first content parameter predictions;
• and
• updating the machine learning model to maximize differences between ground truth content parameters and the first content parameter predictions and differences between ground truth style parameters and the first style parameter predictions.