Systems and methods to predict visemes based on text data and audio frames to facilitate display of animations

A non-transitory, processor-readable medium storing instructions that, when executed by a processor, cause the processor to:

• receive text data;
• provide the text data as input to a first machine learning model to determine an emotion state represented by the text data;
• provide the text data and the emotion state as input to a second machine learning model to generate predicted viseme data;
• generate audio data based on the text data and the emotion state, the audio data representing the text data and defining a clock;
• cause (1) a first portion of the predicted viseme data and (2) a first portion of the audio data to be streamed to a user compute device based on the clock;
• while the first portion of the predicted viseme data and the first portion of the audio data is being streamed to the user compute device, provide the audio data as input to a third machine learning model to generate audio-based viseme data;
• and
• in response to generating the audio-based viseme data:
• cause (1) a portion of the audio-based viseme data and (2) a second portion of the audio data to be streamed to the user compute device based on the clock, and
• exclude a second portion of the predicted viseme data from being streamed to the user compute device, the portion of the audio-based viseme data, the second portion of the audio data, and the second portion of the predicted viseme data being associated with a time specified by the clock.

A method, comprising:

• receiving, via a processor, text data;
• predicting, via the processor, an emotion state based on the text data;
• predicting, via the processor, predicted viseme data based on the text data and the emotion state;
• iteratively generating, via the processor, audio data based on the text data and the emotion state, the audio data (1) representing the text data and defining a clock;
• during the iteratively generating, storing, via the processor, a first portion of the audio data in a buffer;
• in response to the first portion of the audio data stored in the buffer having at least a predefined length, causing, via the processor, (1) a first portion of the predicted viseme data and (2) the first portion of the audio data to be streamed to a user compute device based on the clock;
• while the first portion of the predicted viseme data and the first portion of the audio data is being streamed to the user compute device, generating, via the processor, audio-based viseme data based on the audio data;
• and
• in response to generating the audio-based viseme data:
• causing, via the processor, (1) a portion of the audio-based viseme data and (2) a second portion of the audio data to be streamed to the user compute device based on the clock, and
• excluding, via the processor, a second portion of the predicted viseme data from being streamed to the user compute device, the portion of the audio-based viseme data, the second portion of the audio data, and the second portion of the predicted viseme data being associated with a time specified by the clock.