Back to results
US 7,454,342 B2Granted
Coupled hidden Markov model (CHMM) for continuous audiovisual speech recognition
AssigneeIntel Corporation
InventorsAra Victor Nefian, Xiaoxing Liu, Xiaobo Pi, Luhong Liang, Yibao Zhao
Life of this patent
Estimating the term…
Abstract
Method and apparatus for an audiovisual continuous speech recognition (AVCSR) system using a coupled hidden Markov model (CHMM) are described herein. In one aspect, an exemplary process includes receiving an audio data stream and a video data stream, and performing continuous speech recognition based on the audio and video data streams using a plurality of hidden Markov models (HMMs), a node of each of the HMMs at a time slot being subject to one or more nodes of related HMMs at a preceding time slot. Other methods and apparatuses are also described.
Claims30
A method, comprising:
- receiving an audio data stream and a video data stream;
- and
- performing continuous speech recognition based on the audio and video data streams using a plurality of hidden Markov models (HMMs) including at least one HMM corresponding to each data stream, the HMM having a coupled node, a mixture node and an observable node, the coupled node and the mixture node being hidden, the coupled node at a time slot being subject to a separate coupled node of a separate HMM of the plurality of HMMs at a preceding time slot, the coupled node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the time slot, the mixture node at the time slot being directly subject to the coupled nodes at the time slot, the mixture node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the preceding time slot and the mixture node being not directly subject to any hidden node outside of the HMM.
A method, comprising:
- estimating parameters of a plurality of hidden Markov models (HMMs) individually based on isolated phoneme-viseme pairs of a training sequence using a Viterbi algorithm, the HMMs being based on an audio data stream and a video data stream, the HMMs including at least one HMM corresponding to each data stream, the HMM having a coupled node, a mixture node and an observable node, the coupled node and the mixture node being hidden, the coupled node at a time slot being subject to a separate coupled node of a separate HMM of the HMMs at a preceding time slot, the coupled node at the time slot being not directly subject to any hidden node of the HMMs at the time slot, the mixture node at the time slot being directly subject to the hidden coupled nodes at the time slot, the mixture node at the time slot being not directly subject to any hidden node of the HMMs at the preceding time slot and the mixture node being not directly subject to any hidden node outside of the HMM;
- and
- performing training on the parameters of the HMMs from a continuous audiovisual speech using a Viterbi algorithm.
A computer-readable storage medium having executable codes which when executed by one or more data processing systems cause said systems to perform a method, the method comprising:
- receiving an audio data stream and a video data stream;
- and
- performing continuous speech recognition based on the audio and video data streams using a plurality of hidden Markov models (HMMs) including at least one HMM corresponding to each data stream, the HMM having a coupled node, a mixture node and an observable node, the coupled node and the mixture node being hidden, the coupled node at a time slot being subject to a separate coupled node of a separate HMM of the plurality of HMMs at a preceding time slot, the coupled node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the time slot, the mixture node at the time slot being directly subject to the coupled node at the time slot, the mixture node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the preceding time slot and the mixture node being not directly subject to any hidden node outside of the HMM.
A computer-readable storage medium having executable codes which when executed by one or more data processing systems cause said systems to perform a method, the method comprising:
- estimating parameters of a plurality of hidden Markov models (HMMs) individually based on isolated phoneme-viseme pairs of a training sequence using a Viterbi algorithm, the HMMs being based on an audio data stream and a video data stream, the HMMs including at least one HMM corresponding to each data stream, the HMM having a coupled node, a mixture node and an observable node, the coupled node and the mixture node being hidden, the coupled node at a time slot being subject to a separate coupled node of a separate HMM of the plurality of HMMs at a preceding time slot, the coupled node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the time slot, the mixture node at the time slot being directly subject to the coupled nodes at the time slot, the mixture node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the preceding time slot and the mixture node being not directly subject to any hidden node outside of the HMM;
- and
- performing training on the parameters of the HMMs from a continuous audiovisual speech using a Viterbi algorithm.
A system, comprising:
- a processor;
- and
- a memory coupled to the processor, the memory storing instructions, which when executed by the processor, cause the processor to perform the operations of:
- receiving an audio data stream and a video data stream;
- and
- performing continuous speech recognition based on the audio and video data streams using a plurality of hidden Markov models (HMMs) including at least one HMM corresponding to each data stream, the HMM having a coupled node, a mixture node and an observable node, the coupled node and the mixture node being hidden, the coupled node at a time slot being subject to a separate coupled node of a separate HMM of the plurality of HMMs at a preceding time slot, the coupled node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the time slot, the mixture node at the time slot being directly subject to the coupled nodes at the time slot, the mixture node at the time slot being not directly subject to any hidden node of the plurality of HMMs at the preceding time slot and the mixture node being not directly subject to any hidden node outside of the HMM.
3 more elements
A system, comprising:
- a processor;
- and
- a memory coupled to the processor, the memory storing instructions, which when executed by the processor, cause the processor to perform the operations of:
- estimating parameters of a plurality of hidden Markov models (HMMs) individually based on isolated phoneme-viseme pairs of a training sequence using a Viterbi algorithm, the HMMs being based on an audio data stream and a video data stream, the HMMs including at least one HMM corresponding to each data stream, the HMM having a coupled node, a mixture node and an observable node, the coupled node and the mixture node being hidden, the coupled node at a time slot being subject to a separate coupled node of a separate HMM of the HMMs at a preceding time slot, the coupled node at the time slot being not directly subject to any hidden node of the HMMs at the time slot, the mixture node at the time slot being directly subject to the coupled node at the time slot, the mixture node at the time slot being not directly subject to any hidden node of the HMMs at the preceding time slot and the mixture node being not directly subject to any hidden node outside of the HMM;
- and
- performing training on the parameters of the HMMs from a continuous audiovisual speech using a Viterbi algorithm.
3 more elements
Dependencies are inferred from the claim text.