Speech-driven facial animation generation method

A speech-driven facial animation generation method, comprising steps of:

• (1) extracting speech features:
• extracting Mel spectrogram features from speech in a window;
• the features being a three-dimensional tensor composed of a feature map dimension, a frequency dimension, and a time dimension;
• (2) collecting frequency information:
• for the Mel spectrogram obtained in step (1), abstracting and collecting, by using a neural network, all frequency information along a direction of the frequency dimension, to obtain frequency abstract information;
• (3) summarizing time information;
• for the frequency abstract information obtained in step (2), determining, by using the neural network, a degree of importance of each frame of information in a time context along a direction of the time dimension, and performing summarization according to the degree of importance, to obtain time summary information;
• (4) decoding action features:
• connecting the time summary information obtained in step (3) to a one-hot vector of a control style inputted by a user;
• outputting scaling/shearing coefficients and rotation coefficients respectively through two branches of the neural network, and combining the coefficients outputted by the two branches to obtain deformation gradients representing facial actions;
• (5) driving a facial model:
• for any given expressionless and mouth-closed facial model, driving the facial model by using the deformation gradients obtained in step (4), to perform a corresponding facial action;
• and
• (6) sliding a signal window:
• repeating step (1) to step (5), and processing all speech signal windows, to generate complete facial animation.