Animated speech refinement using machine learning

A system comprising:

• at least one processor configured to perform operations comprising:
• processing, in a training phase for a machine learning model, a plurality of text files by a text-to-speech network to generate a plurality of training synthesized speech streams;
• generating ground truth phoneme timing locations of the plurality of training synthesized speech streams, the ground truth phoneme timing locations generated by the text-to-speech network;
• collecting the plurality of training synthesized speech streams and the ground truth phoneme timing locations into a plurality of training data;
• processing a given training synthesized speech stream of the plurality of training synthesized speech streams by an automated speech recognition (ASR) engine to generate a set of base timing for phonemes of the given training synthesized speech stream, the set of base timings comprising base timing locations generated for the phonemes of the given training synthesized speech stream by the ASR engine;
• applying the machine learning model to the set of base timing for the phonemes of the given training synthesized speech stream to estimate an offset to the set of base timing;
• adjusting the set of base timing using the offset to generate a refined set of base timing for the phonemes;
• comparing the refined set of base timing for the phonemes with the ground truth phoneme timing locations associated with the given training synthesized speech stream to generate a loss;
• updating parameters of the machine learning model based on the loss;
• receiving an audio stream comprising one or more spoken words to control animation of an avatar;
• processing the audio stream by the ASR engine to identify a list of timestamps for a plurality of phonemes of the one or more spoken words;
• providing the list of timestamps to the machine learning model as an input to generate one or more adjustments to one or more timestamps of one or more phonemes;
• updating the one or more timestamps of the one or more phonemes based on the one or more adjustments generated by the machine learning model;
• generating a table comprising the list of timestamps with the updated one or more timestamps, each timestamp in the list of timestamps with the updated one or more timestamps corresponding to a phoneme of the plurality of phonemes, an image, and a gesture;
• and
• generating the animation of the avatar based on the table.

A non-transitory machine-readable storage medium that includes instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

• processing, in a training phase for a machine learning model, a plurality of text files by a text-to-speech network to generate a plurality of training synthesized speech streams;
• generating ground truth phoneme timing locations of the plurality of training synthesized speech streams, the ground truth phoneme timing locations generated by the text-to-speech network;
• collecting the plurality of training synthesized speech streams and the ground truth phoneme timing locations into a plurality of training data;
• processing a given training synthesized speech stream of the plurality of training synthesized speech streams by an automated speech recognition (ASR) engine to generate a set of base timing for phonemes of the given training synthesized speech stream, the set of base timings comprising base timing locations generated for the phonemes of the given training synthesized speech stream by the ASR engine;
• applying the machine learning model to the set of base timing for the phonemes of the given training synthesized speech stream to estimate an offset to the set of base timing;
• adjusting the set of base timing using the offset to generate a refined set of base timing for the phonemes;
• comparing the refined set of base timing for the phonemes with the ground truth phoneme timing locations associated with the given training synthesized speech stream to generate a loss;
• updating parameters of the machine learning model based on the loss;
• receiving an audio stream comprising one or more spoken words to control animation of an avatar;
• processing the audio stream by the ASR engine to identify a list of timestamps for a plurality of phonemes of the one or more spoken words;
• providing the list of timestamps to the machine learning model as an input to generate one or more adjustments to one or more timestamps of one or more phonemes;
• updating the one or more timestamps of the one or more phonemes based on the one or more adjustments generated by the machine learning model;
• generating a table comprising the list of timestamps with the updated one or more timestamps, each timestamp in the list of timestamps with the updated one or more timestamps corresponding to a phoneme of the plurality of phonemes, an image, and a gesture;
• and
• generating the animation of the avatar based on the table.