Late integration in audio-visual continuous speech recognition
AssigneeInternational Business Machines Corporation
InventorsAshish Verma, Sankar Basu, Chalapathy Neti
Life of this patent
Estimating the term…
Abstract
The combination of audio and video speech recognition in a manner to improve the robustness of speech recognition systems in noisy environments. Contemplated are methods and apparatus in which a video signal associated with a video source and an audio signal associated with the video signal are processed, the most likely viseme associated with the audio signal and video signal is determined and, thereafter, the most likely phoneme associated with the audio signal and video signal is determined.
Claims27
A method of providing speech recognition, the method comprising the steps of:
- processing a video signal associated with a video source;
- processing an audio signal associated with the video signal;
- determining the most likely viseme associated with the audio signal and video signal;
- thereafter determining a restricted set of at least one phoneme candidate, the at least one phoneme candidate being related to the most likely viseme;
- and
- thereafter determining, solely from the restricted set of at least one phoneme candidate, the most likely phoneme associated with the audio signal and video signal.
The method according to claim 3, wherein a pair of separate weights are assigned to the video probability of viseme i and the audio probability of viseme i.
Apparatus for providing speech recognition, the apparatus comprising:
- a video signal processor associated with a video source;
- an audio signal processor associated with the video signal;
- a viseme determinator which determines the most likely viseme associated with the audio signal and video signal;
- and
- a phoneme determinator which thereafter determines a restricted set of at least one phoneme candidate, the at least one phoneme candidate being related to the most likely viseme, and thereafter determines, solely from the restricted set of at least one phoneme candidate, the most likely phoneme associated with the audio signal and video signal.
A program storage device readable by machine, tangibly embodying a program of instructions executable by the machine to perform method steps for providing speech recognition, said method steps comprising:
- processing a video signal associated with a video source;
- processing an audio signal associated with the video signal;
- determining the most likely viseme associated with the audio signal and video signal;
- thereafter determining a restricted set of at least one phoneme candidate, the at least one phoneme candidate being related to the most likely viseme;
- and
- thereafter determining, solely from the restricted set of at least one phoneme candidate, the most likely phoneme associated with the audio signal and video signal.
Dependencies are inferred from the claim text.