US 6,250,928 B1Granted

Talking facial display method and apparatus

AssigneeMassachusetts Institute of Technology

InventorsTomaso A. Poggio, Antoine F. Ezzat

Life of this patent

Estimating the term…

Loading patent…

Abstract

A method and apparatus of converting input text into an audio-visual speech stream resulting in a talking face image enunciating the text. This method of converting input text into an audio-visual speech stream comprises the steps of: recording a visual corpus of a human-subject, building a viseme interpolation database, and synchronizing the talking face image with the text stream. In a preferred embodiment, viseme transitions are automatically calculated using optical flow methods, and morphing techniques are employed to result in smooth viseme transitions. The viseme transitions are concatenated together and synchronized with the phonemes according to the timing information. The audio-visual speech stream is then displayed in real time, thereby displaying a photo-realistic talking face.

Claims41

Dependent claims

A method of converting input text into an audio-visual speech stream comprising a talking face image enunciating the text, wherein said audio-visual speech stream comprises a plurality of phonemes and timing information, wherein the tallking face image is built using a plurality of visemes, the method comprising the steps of:

recording a visual corpus of a human-subject;
extracting and defining a plurality of visemes from the recorded visual corpus, said visemes being defined by a set of images spanning a range of mouth shapes derived from the recorded visual corpus;
building a viseme interpolation database, said database comprising a plurality of viseme images and at least one set of interpolation vectors that define a transition from each viseme image to every other viseme image, said viseme images in said interpolation database being a subset of said plurality of visemes extracted from said visual corpus, said set of interpolation vectors being computed automatically (i, in the absence of a definition of a set of high-level features and (ii) through the use of optical flow methods, said viseme interpolation database further comprising a corresponding set of intermediate viseme images automatically generated as a function of respective interpolation vectors;
and
synchronizing the talking face image with an input text stream by employing said interpolation vectors and viseme images contained in said interpolation database, said synchronizing resulting in giving the impression of a photo-realistic talking face.

2 more elements

A system for generating and displaying a talking facial display comprising:

a computer;
an image source in electrical communication with the computer to transfer input images of a human-subject to the computer, the input images composing a visual corpus;
a text data source in electrical communication with the computer to transfer input text to the computer, the input text composing a text stream;
and
processor routines executed by the computer, the processor routines comprising instructions to:
(i) build a viseme interpolation database, said database comprising a plurality of viseme images and at least one set of interpolation vectors that define a transition from each viseme image to every other viseme image, said viseme images in said interpolation database being a subset of a plurality of visemes extracted from said visual corpus, said set of interpolation vectors being computed automatically (i) in the absence of a definition of a set of high-level features and (ii) through the use of optical flow methods, said viseme interpolation database further comprising a corresponding set of intermediate viseme images automatically generated as a function of respective interpolation vectors;
and
(ii) synchronize an image of a talking face with the text stream by employing said interpolation vectors and viseme images contained in said interpolation database, said synchronizing resulting in giving the impression of a photo-realistic talking face.

5 more elements

Dependencies are inferred from the claim text.