Method and apparatus for presenting images representative of an utterance with corresponding decoded speech

Apparatus for presenting images representative of one or more words in an utterance with corresponding decoded speech, the apparatus comprising:

• an automatic speech recognition (ASR) engine for converting the utterance into one or more decoded words, the ASR engine generating time information associated with each of the decoded words;
• a visual detector, the visual detector capturing images of body movements corresponding to one or more words in the utterance;
• a visual feature extractor coupled to the visual detector, the visual feature extractor receiving the time information from the ASR engine and operatively processing the captured images into one or more image segments based on the time information relating to the decoded words, each image segment comprising a plurality of successive images in time corresponding to a decoded word in the utterance;
• and
• an image player operatively coupled to the visual feature extractor, the image player receiving and presenting each image segment with the corresponding decoded word.

A method for presenting images representative of one or more words in an utterance with corresponding decoded speech, the method comprising the steps of:

• capturing a plurality of images representing body movements corresponding to the one or more words in the utterance;
• associating each of the captured images with time information relating to an occurrence of the image;
• receiving, from an automatic speech recognition (ASR) system, data including a start time and an end time of a word decoded by the ASR system;
• aligning the plurality of images into one or more image segments according to the start and stop times received from the ASR system, wherein each image segment corresponds to a decoded word in the utterance;
• and
• presenting an image segment with a corresponding decoded word.

In an automatic speech recognition (ASR) system for converting an utterance of a speaker into one or more decoded words, a method for enhancing the ASR system comprising the steps of:

• capturing a plurality of successive images in time representing body movements corresponding to one or more words in the utterance;
• associating each of the captured images with time information relating to an occurrence of the image;
• obtaining, from the ASR system, time ends for each decoded word in the utterance;
• grouping the plurality of images into one or more image segments based on the time ends, wherein each image segment corresponds to a decoded word in the utterance;
• and
• presenting an image segment with a corresponding decoded word.

A method for presenting images representative of one or more words in an utterance with corresponding decoded speech, the method comprising the steps of:

• providing an automatic speech recognition (ASR) engine;
• decoding, in the ASR engine, the utterance into one or more words, each of the decoded words having a start time and a stop time associated therewith;
• capturing a plurality of images representing body movements corresponding to the one or more words in the utterance;
• buffering the plurality of images by a predetermined delay;
• receiving, from the ASR engine, data including the start time and the end time of a decoded word;
• aligning the plurality of images into one or more image segments according to the start and stop times received from the ASR engine, wherein each image segment corresponds to a decoded word in the utterance;
• and
• presenting an image segment with a corresponding decoded word.