Generating expressive facial animation data from speech audio using speech emotion recognition

A computing system for generating facial animation data that animates a face in a video game in accordance with speech sounds and emotion of speech audio, the computing system comprising one or more computing devices configured to:

• generate one or more emotion representations that represent the emotion of the speech audio, comprising processing, using a machine-learned speech emotion recognition model, speech audio data representing the speech audio;
• wherein each of the one or more emotion representations comprises a respective score for each emotion classification of a set of emotion classifications;
• wherein each emotion classification of the set of emotion classifications is associated with a pre-determined conditioning input;
• determine, for each of the one or more emotion representations, a respective conditioning input for a facial animation generative model comprising:
• for each emotion classification of the set of emotion classifications, weighting the pre-determined conditioning input associated with the emotion classification based on the respective score for the emotion classification generated by the machine-learned speech emotion recognition model;
• and
• summing together the weighted pre-determined conditioning inputs to generate the respective conditioning input for the facial animation generative model;
• and
• generate facial animation data for the speech audio data, comprising processing, using the facial animation generative model:
• (i) input data derived from the speech audio, and (ii) the one or more conditioning inputs.

A non-transitory computer-readable medium storing instructions, which when executed by one or more processors, cause the processor to:

• generate one or more emotion representations for speech audio, comprising processing, using a machine-learned speech emotion recognition model, speech audio data representing the speech audio;
• wherein each of the one or more emotion representations comprises a respective score for each emotion classification of a set of emotion classifications;
• wherein each emotion classification of the set of emotion classifications is associated with a pre-determined conditioning input;
• determine, for each of the one or more emotion representations, a respective conditioning input for a facial animation generative model comprising:
• for each emotion classification of the set of emotion classifications, weighting the pre-determined conditioning input associated with the emotion classification based on the respective score for the emotion classification generated by the machine-learned speech emotion recognition model;
• and
• summing together the weighted pre-determined conditioning inputs to generate the respective conditioning input for the facial animation generative model;
• and
• generate facial animation data for the speech audio data, comprising processing, using the facial animation generative model:
• (i) input data derived from the speech audio, and (ii) the one or more conditioning inputs.