Speech detection and enhancement using audio/video fusion

#The present invention relates generally to signal enhancement, and more particularly to a system and method facilitating speech detection and/or enhancement through a probabilistic-based model that fuses audio and video fusion.

#The ease with which individuals can carry on a conversation in the midst of noise is often taken for granted. Sounds from different sources coalesce and obscure each other making it difficult to resolve what is heard into its constituent parts, and identify its source and content. This auditory scene analysis problem confounds current automatic speech recognition systems, which can fail to recognize speech in the presence of very small amounts of interfering noise. With regard to humans, vision often plays a crucial role, because individuals often have an unobstructed view of the lips that modulate the sound. In fact lip-reading can enhance speech recognition in humans as much as removing 15 dB of noise. This fact has motivated efforts to use video information for tasks of audio-visual scene analysis, such as speech recognition and speaker detection. Such systems have typically been built using separate modules for tasks such as tracking the lips, extracting features, and detecting speech components, where each module is independently designed to be invariant to different speaker characteristics, lighting conditions, and noise conditions.

#One problem with modular systems designed for a variety of conditions is that there is typically a tradeoff between average performance across conditions and performance in any one condition. Thus, for example, a system that can adapt to a face under the current lighting condition may perform better than one designed for a variety of conditions without adaptation. Another pitfall of modular audio-visual systems is that the modules may be integrated in an ad hoc way that neglects information about the uncertainty within models, as well as neglecting statistical dependencies between the modalities. The two problems are related in that unsupervised adaptation is greatly facilitated by enforcing agreement between the audio and video modules during adaptation.

#FIG. 1 is a block diagram of a system that facilitates enhancement of a speech signal in accordance with an aspect of the present invention.

#FIG. 2 is a graphical model representation of a generative model for audio in accordance with an aspect of the present invention.

#FIG. 3 is a graphical model representation of a generative model for video in accordance with an aspect of the present invention.

#The present invention is now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. It may be evident, however, that the present invention may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing the present invention.

#As used in this application, the term “computer component” is intended to refer to a computer-related entity, either hardware, a combination of hardware and software, software, or software in execution. For example, a computer component may be, but is not limited to being, a process running on a processor, a processor, an object, an executable, a thread of execution, a program, and/or a computer. By way of illustration, both an application running on a server and the server can be a computer component. One or more computer components may reside within a process and/or thread of execution and a component may be localized on one computer and/or distributed between two or more computers.

#Referring to FIG. 1, a system 100 that facilitates enhancement of a speech signal in accordance with an aspect of the present invention is illustrated. The system 100 fuses audio and video in a probabilistic generative model that implements cross-model, self-supervised learning, enabling rapid adaptation to audio visual data. The system 100 can learn to detect and enhance speech in noise given only a short (e.g., 30 second) sequence of audio-visual data. Further, in one example, the system 100 automatically learns to track the lips as they move around in the video.