Generating realistic facial animation from speech

A method for generating realistic facial animation directly from speech comprising the steps of:

• providing video frame images and a synchronized audio file of an individual talking;
• entropically training a facial hidden Markov model on said video so as to induce a facial state machine having facial states and the mapping of said facial states to facial configurations;
• estimating mappings from said facial states to audio features to provide a vocal hidden Markov model having facial dynamics, but driven by audio-to-audio sequences;
• predicting facial states from new audio using said vocal hidden Markov model to provide an optimal facial state sequence;
• combining said facial state sequence with said facial mappings to provide an optimal trajectory through said facial configuration space;
• and,
• generating a sequence of animation control parameters therefrom.

A method for generating facial animation, comprising:

• identifying, from a plurality of first audio features which are mapped to associated first facial states, those of the first audio features which correspond to second audio features;
• identifying the first facial states associated with the identified first audio features;
• and
• generating a facial animation corresponding to the second audio features using the identified first facial state;
• wherein the identifying of the first facial states includes identifying a sequence of the predicted first facial states, and identifying a trajectory through the identified sequence of predicted first facial states;
• and
• wherein the identified first facial states are those of the predicted first facial states within the identified sequence which are along the identified trajectory.

A method for generating facial animation, comprising:

• identifying, from a plurality of first audio features which are mapped to associated first facial states, each of the first facial states corresponding to a facial expression, those of the first audio features which correspond to second audio features;
• identifying the first facial states associated with the identified first audio features;
• identifying average facial expressions from the facial expressions corresponding to the identified first facial states;
• and
• generating a facial animation, using the identified first facial state, which corresponds to the second audio features and represents the identified average facial expressions.

A method for generating facial animation, comprising:

• identifying, from a plurality of first audio features which are mapped to associated first facial states, those of the first audio features which correspond to second audio features;
• identifying the first facial states associated with the identified first audio features;
• generating a facial animation corresponding to the second audio features using the identified first facial state;
• generating a facial hidden Markov model representing the first facial states;
• and
• generating a vocal hidden Markov model representing a mapping of the first facial states with the associated first audio features;
• wherein the first audio features which correspond to the second audio features are identified using the vocal hidden Markov model and the first facial states associated with the identified first audio features are identified using the facial hidden Markov model.

A method for generating facial animation, comprising:

• identifying, from a plurality of first audio features which are mapped to associated first facial states, those of the first audio features which correspond to second audio features;
• identifying the first facial states associated with the identified first audio features;
• and
• generating a facial animation corresponding to the second audio features using the identified first facial state;
• wherein the first facial states represent contextual information associated the first audio features.

A method for generating facial animation, comprising:

• identifying, from a plurality of first audio features which are associated with a training audio and are mapped to associated first facial states, those of the first audio features which correspond to second audio features;
• identifying the first facial states associated with the identified first audio features;
• generating a facial animation corresponding to the second audio features using the identified first facial state;
• tracking facial features presented in a training video synchronized to the training audio;
• generating the first facial states using the tracked facial features;
• and
• determining which of the tracked facial features would most probably have been presented in the training video if the training audio included the second audio features;
• wherein the identified first facial states are identified based upon the tracked facial features determined to most probably have been presented.

An article of manufacture for generating facial animation, comprising:

• a computer readable storage medium;
• and
• computer programming stored on the medium, wherein the stored computer programming is configured to be readable from the medium by a computer and thereby cause the computer to operate so as to:
• identify facial poses and associated facial dynamics presented in a training video, which correspond to first audio features presented in a training audio synchronized to the training video;
• generate a facial hidden Markov model representing the facial dynamics mapped to the associated facial poses;
• and
• generate a vocal hidden Markov model representing the facial poses mapped to the corresponding first audio features;
• wherein second audio features which correspond to the first audio features are mappable to the corresponding first facial poses using the vocal hidden Markov model and the first poses mapped to the corresponding second audio features are mappable to the associated facial dynamic using the facial hidden Markov model to generate a facial animation.

An article of manufacture for generating facial animation, comprising:

• a computer readable storage medium;
• and
• computer programming stored on the medium, wherein the stored computer programming is configured to be readable from the medium by a computer and thereby cause the computer to operate so as to:
• identify first audio features from a non-phonemic training audio representation which correspond to second audio features of a new non-phonemic audio representation;
• identify a sequence of predicted facial states from a training video associated with the identified first audio features;
• identify a trajectory through the identified sequence of predicted facial states;
• identify facial states within the identified sequence of predicted facial states along the identified trajectory;
• and
• generate a facial animation corresponding to the second audio features using the identified facial states.

A method for generating facial animation, comprising the steps of:

• identifying facial poses and associated facial dynamics presented in a training video, which correspond to first audio features presented in a training audio synchronized to the training video;
• generating a facial hidden Markov model representing the facial dynamics mapped to the associated facial poses;
• and
• generating a vocal hidden Markov model representing the facial poses mapped to the corresponding first audio features;
• wherein second audio features which correspond to the first audio features are mappable to the corresponding first facial poses using the vocal hidden Markov model and the first poses mapped to the corresponding second audio features are mappable to the associated facial dynamic using the facial hidden Markov model to generate a facial animation.

A method for generating facial animation, comprising:

• identifying first audio features from a training audio which correspond to second audio features of a new audio;
• identifying a sequence of the predicted facial states from a training video associated with the identified first audio features;
• identifying a trajectory through the identified sequence of predicted facial states;
• identifying facial states within the identified sequence of along the identified trajectory;
• and
• generating a facial animation corresponding to the second audio features using the identified facial states.

A method for generating realistic facial animation directly from an audio signal, comprising:

• learning a model of facial states from observations of a face while a person is generating a training audio signal;
• relating the training audio signal to the facial states;
• generating a sequence of facial states in relation to an input audio signal based on a correspondence of the input audio signal to the training audio signal;
• converting the sequence of facial states into control parameters;
• and
• driving facial animation with the control parameters;
• and
• wherein the model is a hidden Markov model including transition dynamics for controlling transitions between the facial states;
• and
• wherein the transition dynamics include an average duration of each facial state, a likelihood of each potential successor facial state, and information representing potential past and future facial states carried forward and backward in time used for enforcing long-term dependencies between the facial states.