Method and system for converting text to lip-synchronized speech in real time

A system for presenting a lip-syncing character, comprising:

• a rules store containing rules for controlling modification of words of sequence of words, the rules including rules to add a sound after a phrase and to remove certain verbs;
• an expression store containing images of a character representing different expressions of emotion for that character;
• a modify word component that receives a sequence of words in real time and modifies the words of the sequence in accordance with the rules of the rules store;
• an identify expressions component that identifies expressions of emotion from the words of the sequence and maps the expressions of emotion to the words;
• a lip synchronization component that inputs the modified words of the sequence, the map of expressions of emotion to the words, and the images of the character representing different expressions of emotion and outputs in real time as the words are received speech corresponding to the modified words of the sequence and images of the character speaking the output speech and having the identified expressions of emotion synchronized to the speech as indicated by the map and images of hands of the character moving to effect output of the modified words in a sign language.

A computer-readable storage medium containing instructions for controlling a computer to present images of a character speaking, by a method comprising:

• providing a plurality of rules for controlling modification of words of a sequence of words, the rules including rules to add a sound after a phrase and to replace words with words of different complexity;
• providing images of a character representing different expressions of emotion of the character;
• receiving a sequence of words in real time;
• modifying the words of the sequence in accordance with the provided rules;
• after modifying the words, generating speech corresponding to the received sequence of words as modified;
• identifying expressions of emotion from the words of the received sequence of words;
• generating a sequence of images based on the provided images to represent the character speaking the generated speech and exhibiting the identified expressions of emotion so that the speaking of a word is synchronized with an expression of emotion identified from that word and to represent the character using a sign language to effect the output of modified words of the sequence;
• and
• outputting the generated speech and sequence of images to portray the character speaking the text with the identified expression of emotion.