System and method for rendering three dimensional face model based on audio stream and image data

A system comprising:

• a microphone;
• an imaging device;
• one or more processors coupled to the microphone and the imaging device;
• and
• a non-transitory computer readable medium storing instructions when executed by the one or more processors cause the one or more processors to:
• receive, through the microphone, a vocal output of a person over a plurality of time instances, as an audio stream;
• acquire, through the imaging device, images of facial expressions of the person over the plurality of time instances, as corresponding image data;
• predict phonemes of the vocal output according to the audio stream;
• translate the predicted phonemes of the vocal output into visemes;
• determine a plurality of blendshapes and corresponding weights, according to the corresponding image data of the face, to form a three-dimensional (3D) model of an avatar of the person incorporating the facial expressions of the person, the plurality of blendshapes comprising 3D structures, the corresponding weights indicating an amount of transformation applied to the 3D structures of the 3D model of the avatar;
• combine the visemes with the 3D model of the avatar to form a 3D representation of the avatar, by synchronizing the visemes with the 3D model of the avatar over the plurality of time instances;
• and
• render the 3D representation of the avatar with the incorporated facial expressions of the person in synchronization with rendering of the audio stream.

A non-transitory computer readable medium storing instructions when executed by one or more processors cause the one or more processors to:

• receive, through a microphone, a vocal output of a person over a plurality of time instances, as an audio stream;
• acquire, through an imaging device, images of facial expressions of the person over the plurality of time instances, as corresponding image data;
• predict phonemes of the vocal output according to the audio stream;
• translate the predicted phonemes of the vocal output into visemes;
• determine a plurality of blendshapes and corresponding weights, according to the corresponding image data of the face, to form a three-dimensional (3D) model of an avatar of the person incorporating the facial expressions of the person, the plurality of blendshapes comprising 3D structures, the corresponding weights indicating an amount of transformation applied to the 3D structures of the 3D model of the avatar;
• combine the visemes with the 3D model of the avatar to form a 3D representation of the avatar, by synchronizing the visemes with the 3D model of the avatar over the plurality of time instances;
• and
• render the 3D representation of the avatar with the incorporated facial expressions of the person in synchronization with rendering of the audio stream.