Three-dimensional face animation from speech

A computer-implemented method, comprising:

• identifying, from an audio capture of a subject, an audio-correlated facial feature;
• generating a first mesh for a lower portion of a face of the subject, based on the audio-correlated facial feature;
• identifying an expression-like facial feature of the subject;
• generating a second mesh for an upper portion of a face of the subject based on the expression-like facial feature;
• forming a synthesized mesh with the first mesh and the second mesh;
• generating a three-dimensional model of the face of the subject with the synthesized mesh based on a ground truth image of the subject;
• and
• providing the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application that includes the subject.

A system, comprising:

• one or more processors;
• and
• a memory storing instructions which, when executed by the one or more processors, cause the system to:
• identify, from an audio capture of a subject, an audio-correlated facial feature;
• generate a first mesh for a lower portion of a face of the subject, based on the audio-correlated facial feature;
• identify an expression-like facial feature of the subject;
• generate a second mesh for an upper portion of a face of the subject based on the expression-like facial feature;
• form a synthesized mesh with the first mesh and the second mesh;
• generate a three-dimensional model of the face of the subject with the synthesized mesh based a ground truth image of the subject;
• and
• provide the three-dimensional model of the face of the subject to a display in a client device running an immersive reality application that includes the subject.

A computer-implemented method, comprising:

• determining a first correlation value for a facial feature based on an audio waveform from a first subject;
• generating a first mesh for a lower portion of a human face, based on the facial feature and the first correlation value;
• updating the first correlation value based on a difference between the first mesh and a ground truth image of the first subject;
• and
• providing a three-dimensional model of the human face animated by speech to an immersive reality application accessed by a client device based on the difference between the first mesh and the ground truth image of the first.