Method and system for integrated audiovisual speech coding at low bitrate

A face image transmitting system for transmitting a motion picture of a face when a speaker on a transmitting side speaks to a receiving side, comprising:

• a transmitting apparatus transmitting a speech signal produced by the speaker to the receiving side, wherein the transmitting apparatus includes
• a feature extracting means for time sequentially extracting predetermined features of the speech produced by the speaker,
• an artificial neural network capable of learning such that, using feature parameters output from the feature extraction means based on the speech produced by the speaker, information specifying a face image of the speaker is output, and
• a means for transmitting the feature parameters of the artificial neural network to the receiving side;
• and
• a receiving apparatus, wherein the receiving apparatus includes
• a neural network based facial shape estimation means receiving the speech signal spoken by the speaker transmitted from the transmitting apparatus and outputting a signal estimating the facial shape of the speaker when he/she speaks the speech, and
• face image generation means generating the motion picture of the facial shape of the speaker based on the signal estimation the facial shape of the speaker output by the facial shape estimation means upon reception of the speech signal produced by the speaker from the transmitting side;
• wherein said facial shape estimation means includes
• an artificial neural network identical in configuration to the artificial neural network of the transmitting apparatus, and
• means for setting the artificial neural network of the receiving apparatus with the feature parameters transmitted from the transmitting apparatus.

A face image transmitting apparatus used in a face image transmitting system transmitting motion picture of a face when a speaker speaks on a transmitting side to a receiving side, comprising:

• means for transmitting speech produced by the speaker to the receiving side;
• an artificial neural network capable of learning such that information specifying the face image of the speaker is output, using features of the speech produced by the speaker as input;
• and
• means for transmitting the feature parameters of the artificial neural network to the receiving side;
• measuring means for time sequentially measuring coordinates of predetermined portions of the speaker's face;
• feature extracting means for time sequentially extracting predetermined features of the speech spoken by the speaker;
• and
• means for implementing training of the artificial neural network using the features obtained by the feature extracting means when the speaker reads training sentences as input and the coordinates measured by the measuring means when the training sentences are read as a teacher signal.

A face image recovering apparatus, comprising:

• a neural network based facial shape estimation means receiving predetermined features obtained from the speech produced by the speaker, for estimating facial shape of the speaker when he/she produces speech;
• and
• face image generation means for generating a motion picture of the face image of the speaker based on the signal estimating the facial shape of the sneaker output by the facial shape estimation means upon reception of the features, wherein the facial shape estimation means includes
• an artificial neural network of a predetermined configuration, and
• means, receiving feature parameters for setting the artificial neural network before recovery of a face image from a predetermined signal source, for setting the artificial neural network with the feature parameters.

A face image transmitting system for transmitting a face of a speaker on the transmitting side when he/she speaks to the receiving side, comprising:

• a transmitting apparatus transmitting the speech signal spoken by the speaker to the receiving side, wherein the transmitting apparatus includes
• a facial shape estimation circuit estimating the facial shape of the speaker by time sequentially extracting predetermined feature parameters of the speech produced by the speaker,
• an artificial neural network capable of learning such that information specifying the face image of the speaker is output, using the feature parameters output from the facial shape estimation circuit as input, and
• a transmitting circuit transmitting the feature parameters of the artificial neural network to the receiving side;
• and
• a receiving apparatus that includes
• a neural network based facial shape estimation circuit receiving the speech signal spoken by the speaker transmitted from the transmitting apparatus and outputting a signal associating the speech with an estimated facial shape of the speaker when he/she speaks, wherein the facial shape estimation circuit includes an artificial neural network identical in configuration to the artificial neural network of the transmitting apparatus and set by the feature parameters transmitted from the transmitting apparatus, and
• a face image generation circuit generating a motion picture of the facial shape of the speaker based on the signal output by the facial shape estimation circuit upon reception of the speech signal produced by the speaker from the transmitting side.

A face image transmitting apparatus used in a face image transmitting system for transmitting motion picture of the face of a speaker on a transmitting side when he/she speaks to a receiving side, comprising:

• a speech signal transmission circuit transmitting the speaker's speech to a receiving apparatus;
• an artificial neural network capable of learning such that information specifying the face image of the speaker is output, using features of the speech produced by the speaker as input;
• a parameter transmitting circuit transmitting feature parameters of the artificial neural network to the receiving side;
• a measurement circuit measuring coordinates of predetermined portions of the speaker's face in a time sequence;
• a speech parameter matrix generation circuit time sequentially extracting predetermined feature parameters of the speech spoken by the speaker and generating a speech parameter matrix;
• and
• a parameter training circuit implementing training of the artificial neural network, using the speech parameter matrix generated by the speech parameter matrix generation circuit when the speaker reads training sentences as input, and using the coordinates measured by the measuring circuit when the training sentences are read as a teacher signal.

A face image recovering apparatus, comprising:

• a neural network based facial shape estimation circuit receiving predetermined features obtained from the speech produced by the speaker, and outputting a signal estimating the facial shape of the speaker when the speaker produces speech, wherein the facial shape estimation circuit includes
• an artificial neural network of predetermined configuration, and
• a circuit receiving feature parameters for setting the artificial neural network from a predetermined signal source before recovering a face image, and setting the artificial neural network by the feature parameters;
• and
• a face image generation circuit for generating a motion picture of the facial shape of the speaker based on the signal estimating the facial shape of the speaker output by the facial shape estimation circuit upon reception of the feature.

A method of generating a face image, comprising the following steps:

• preparing a facial shape estimation means receiving speech produced by a speaker, and outputting a signal estimating the facial shape of the speaker during speech production;
• receiving the speech produced by the speaker, applying the received speech to the facial shape estimation means and obtaining a signal estimating facial shape of the speaker based on a neural network technique;
• and
• generating a motion picture of the facial shape of the speaker based on the signal estimating the facial shape of the speaker output from the facial shape estimation means;
• wherein the step of preparing includes the steps of
• preparing artificial neural networks identical to each other in the transmitting side and the receiving side,
• performing a training process on the artificial neural networks such that information specifying a face image of the speaker is output upon reception of the speech produced by the speaker, and
• receiving feature parameters of the trained artificial neural network and setting the artificial neural networks of the receiving side with the transmitted feature parameters.