Methods and devices for producing and using synthetic visual speech based on natural coarticulation

A method of generating synthetic visual speech, comprising:

• receiving a voice input including speech information;
• classifying the voice input into phonemes using a phoneme neural network;
• identifying a viseme corresponding to each of the phonemes from the phoneme neural network;
• calculating a viseme track for each of the visemes using a viseme neural network, wherein the viseme track comprises a sequence of viseme weights over time, and wherein each viseme weight represents an amount of influence of the viseme over other visemes active at a given time;
• morphing between target models of the visemes according to their tracks by producing a series of blended models;
• and
• rendering the series of blended models sequentially to produce a visual speech animation.

A computer readable medium storing computer code comprising:

• instructions for receiving an input including speech information;
• instructions for identifying visemes corresponding to the input;
• instructions for calculating a weight of each of the visemes corresponding to the input using deformability information, wherein each of the viseme weights corresponds to an amount of influence that the viseme has over other visemes active at a given time;
• and
• instructions for producing a synthetic visual speech output based on the weights of the visemes corresponding to the input.

A system for producing synthetic visual speech, comprising:

• a receiver to receive an input representing a speech segment;
• a first neural network to classify the speech segment according to its phonetic components;
• a coarticulation engine comprising deformability information to determine viseme tracks corresponding to the phonetic components of the speech input, wherein the viseme tracks each comprise a sequence of viseme weights over time, and wherein each viseme weight represents an amount of influence of the viseme over other visemes active at a given time;
• and
• a morphing engine for morphing between viseme models based on the viseme tracks to enable a realistic synthetic visual speech output corresponding to the speech segment.

A coarticulation engine for calculating viseme tracks comprising:

• a coarticulation algorithm configured to receive data inputs corresponding to a plurality of visemes;
• said data inputs representing a context, and a duration of each of the visemes;
• and
• said coarticulation engine further configured to produce data outputs comprising a weight for each of the visemes using deformability information, wherein each viseme weight represents an amount of influence of the viseme over other visemes active at a given time.

A method for generating a user-customizable three-dimensional lipsyncing greeting card, comprising:

• receiving a user-defined input containing speech information;
• converting the input into a customized electronic greeting card comprising a three-dimensional visual speech animation comprising a lipsyncing character synchronized with an audio output corresponding to the input;
• and
• delivering the customized electronic greeting card to a recipient identified by the user.

A method of providing an electronic greeting card featuring a three-dimensional lipsyncing character, comprising:

• providing an Internet site;
• allowing a user to supply an input containing speech information to the internet site;
• converting the user-supplied input into an electronic greeting card comprising a three-dimensional lipsyncing character animated in synchronism with an audio output corresponding to the input;
• and
• delivering the electronic greeting card to a recipient specified by the user.

A method for producing a computer animated lipsyncing, comprising:

• providing a voice input into a first neural network to produce a phoneme output;
• providing the phoneme output from the first neural network to a second neural network to produce a viseme track output, wherein the viseme track output comprises a sequence of viseme weights over time, and wherein each viseme weight represents an amount of influence of the viseme over other visemes active at a given time;
• and
• using the viseme track output to generate an animated three-dimensional lipsyncing image in real-time in substantial synchronism with an audio speech output corresponding to the voice input.

An apparatus for producing a lipsyncing animation, comprising:

• a frame processor to identify frames of a voice input;
• a first neural network to receive the frames of the voice input and to identify a probable phoneme corresponding to each of the frames;
• a second neural network to receive the probable phonemes and identify viseme weights for one or more visemes active during each of the frames, wherein a viseme weight represents an amount of influence of a corresponding viseme over other visemes active during that frame;
• and
• a rendering engine to render a three-dimensional lipsyncing animation based on the viseme weights in substantial synchronization with an audio output corresponding to the voice input.

A method for producing a synthesized visual communication over a network comprising:

• receiving an input containing speech information into a first networked device;
• converting the input into phonetic speech components using a phoneme neural network;
• converting the phonetic speech components into weighted visual speech information, wherein the weighted visual speech information comprises information representing an amount of influence of a visual speech component over other visual speech components active at a given time;
• producing a lipsyncing animation based on the weighted visual speech information;
• and
• displaying the lipsyncing animation in substantial synchronism with an audibilization of a voice output corresponding to the input through a second networked device.

A method for providing a real-time synthetic communication comprising:

• providing inputs containing speech information into a first one or more of a plurality of devices;
• converting the inputs into viseme tracks, wherein the viseme tracks each comprise a sequence of viseme weights over time, and wherein each viseme weight represents an amount of influence of the viseme over other visemes active at a given time;
• producing a communication comprising a synthesized visual speech animation for each of the inputs based on the viseme tracks, said communication further comprising an audio output corresponding to the input;
• and
• outputting the communication through a second one or more of the devices.

An email reader comprising:

• a phoneme neural classifier for converting an email text or an audio attachment into its constituent plurality of phonemes;
• a coarticulation engine to determine a weight of each of a plurality of visemes associated with each of the phonemes, wherein each viseme weight represents an amount of influence of the corresponding viseme over other visemes active at a given time;
• a morphing engine for morphing between target viseme models based on viseme weights;
• a text-to-audio speech synthesizer for synthesizing an audio voice output based on the phonemes from the email text;
• and
• a rendering engine for rendering an email lipsyncing animation based on data from the morphing engine.