# US 7,613,613 B2 · Method and system for converting text to lip-synchronized speech in real time

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 7,613,613 B2
- Document ID: US7613613B2
- Status: Granted
- Kind: B2
- Application number: 11/009,966
- Filed: 2004-12-10
- Published: 2006-06-15
- Granted: 2009-11-03
- Earliest claimed: 2004-12-10
- Assignees: Microsoft Corporation
- Inventors: Timothy V. Fields; Brandon Cotton
- Web page: https://patentgrep.com/patent/US7613613B2

## CPC

- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 21/06: Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids (G10L15/26 takes precedence)
- H04N 5/278: Subtitling
- H04N 21/440236: {by media transcoding, e.g. video is transformed into a slideshow of still pictures, audio is converted into text}
- G10L 2021/105: {Synthesis of the lips movements from speech, e.g. for talking heads}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

A method and system for presenting lip-synchronized speech corresponding to the text received in real time is provided. A lip synchronization system provides an image of a character that is to be portrayed as speaking text received in real time. The lip synchronization system receives a sequence of text corresponding to the speech of the character. It may modify the received text in various ways before synchronizing the lips. It may generate phonemes for the modified text that are adapted to certain idioms. The lip synchronization system then generates the lip-synchronized images based on the phonemes generated from the modified texts and based on the identified expressions.

## Claims

### Claim 1

A method for presenting information in real time, the method comprising:
providing a plurality of rules for controlling modification of words of a sequence of words, the rules including rules to add a sound after a phrase, to replace words with words of different complexity, to remove certain verbs without replacing the verbs, and to modify words based on identification of a current expression derived from comparison of words of the sequence to be spoken;
providing an expression store with images of a character representing different expressions of emotion for that character;
receiving a sequence of words;
modifying the words of the received sequence by for each of a plurality of rules,
determining whether the rule applies to words of the received sequence; and
when it is determined that the rule applies, modifying the words of the received sequence in accordance with the rule;
generating speech for the character corresponding to the modified words, the speech represented by a sequence of phonemes including replacing phonemes with other phonemes to achieve regional effects;
identifying expressions of emotion from the words of the received sequence;
mapping the phonemes of the speech and the identified expressions for the character to the words of the received sequence;
generating a sequence of images based on the images of the expression store to represent the character speaking the generated speech and having the identified expressions of emotion and to represent hands of the character moved to effect output of the modified words in a sign language, wherein the mapping to words of the received sequence is used to synchronize the movement of the lips representing the character enunciating the phonemes of the words with the image of the character exhibiting the identified expressions of emotion mapped to those words so that the speaking of a word is synchronized with the image of the character exhibiting the expression of emotion identified from that word; and
outputting the generated speech represented by the sequence of phonemes and the sequence of generated images to portray the character speaking the words of the modified received sequence and having the identified expressions.

### Claim 2 (depends on 1)

The method of claim 1 wherein the sequence of words is closed-captioned text of a television broadcast.

### Claim 3 (depends on 1)

The method of claim 1 wherein the sequence of words is entered via a keyboard by a participant in a computer-based chat session.

### Claim 4 (depends on 1)

The method of claim 1 wherein a rule specifies to modify the words of the sequence by expanding acronyms.

### Claim 5 (depends on 1)

The method of claim 1 wherein a rule specifies to modify the words of the sequence to reflect an idiom.

### Claim 6 (depends on 5)

The method of claim 5 wherein the idiom is associated with the character.

### Claim 7 (depends on 1)

The method of claim 1 wherein the generating of speech includes identifying phonemes from the modified words.

### Claim 8 (depends on 7)

The method of claim 7 wherein the phonemes are identified to reflect an idiom.

### Claim 9

A system for presenting a lip-syncing character, comprising:
a rules store containing rules for controlling modification of words of sequence of words, the rules including rules to add a sound after a phrase and to remove certain verbs;
an expression store containing images of a character representing different expressions of emotion for that character;
a modify word component that receives a sequence of words in real time and modifies the words of the sequence in accordance with the rules of the rules store;
an identify expressions component that identifies expressions of emotion from the words of the sequence and maps the expressions of emotion to the words;
a lip synchronization component that inputs the modified words of the sequence, the map of expressions of emotion to the words, and the images of the character representing different expressions of emotion and outputs in real time as the words are received speech corresponding to the modified words of the sequence and images of the character speaking the output speech and having the identified expressions of emotion synchronized to the speech as indicated by the map and images of hands of the character moving to effect output of the modified words in a sign language.

### Claim 10 (depends on 9)

The system of claim 9 wherein the sequence of words is closed-captioned text of a television broadcast.

### Claim 11 (depends on 9)

The system of claim 9 wherein the sequence of words is entered via a keyboard by a participant in a computer-based chat session.

### Claim 12 (depends on 9)

The system of claim 9 wherein a rule specifies to modify the words of sequence by expanding acronyms.

### Claim 13 (depends on 9)

The system of claim 9 wherein a rule specifies to modify the words of the sequence to reflect an idiom.

### Claim 14 (depends on 9)

The system of claim 9 wherein the generating of speech includes identifying phonemes from the modified words of the sequence.

### Claim 15 (depends on 14)

The system of claim 14 wherein the phonemes are identified to reflect an idiom.

### Claim 16

A computer-readable storage medium containing instructions for controlling a computer to present images of a character speaking, by a method comprising:
providing a plurality of rules for controlling modification of words of a sequence of words, the rules including rules to add a sound after a phrase and to replace words with words of different complexity;
providing images of a character representing different expressions of emotion of the character;
receiving a sequence of words in real time;
modifying the words of the sequence in accordance with the provided rules;
after modifying the words, generating speech corresponding to the received sequence of words as modified;
identifying expressions of emotion from the words of the received sequence of words;
generating a sequence of images based on the provided images to represent the character speaking the generated speech and exhibiting the identified expressions of emotion so that the speaking of a word is synchronized with an expression of emotion identified from that word and to represent the character using a sign language to effect the output of modified words of the sequence; and
outputting the generated speech and sequence of images to portray the character speaking the text with the identified expression of emotion.

### Claim 17 (depends on 16)

The computer-readable medium of claim 16 wherein the sequence of words is closed-captioned text.

### Claim 18 (depends on 16)

The computer-readable medium of claim 16 wherein the sequence of words is entered by a participant in a computer-based chat session.

### Claim 19 (depends on 16)

The computer-readable medium of claim 16 wherein a rule specifies to modify the words of the sequence by expanding acronyms.

### Claim 20 (depends on 16)

The computer-readable medium of claim 16 wherein a rule specifies to modify the words of the sequence to reflect an idiom.

### Claim 21 (depends on 16)

The computer-readable medium of claim 16 wherein the generating of speech includes identifying phonemes from the words of the sequence.

### Claim 22 (depends on 21)

The computer-readable medium of claim 21 wherein the phonemes are identified to reflect an idiom.

### Claim 23 (depends on 16)

The computer-readable medium of claim 16 wherein different images of the character are provided for different expressions.

### Claim 24 (depends on 16)

The computer-readable medium of claim 16 wherein the generating of the sequence of images represents the character lip-syncing the generated speech.