Noise reduction and audio-visual speech activity detection

A near-end speaker detection method for reducing noise in a detected analog audio sequence, said method comprising:

• converting said analog audio sequence into a digital, audio sequence;
• calculating a corresponding discrete signal spectrum of the digital audio sequence by performing a Fast Fourier Transform (FFT);
• detecting a voice of a speaker from said discrete signal spectrum by analyzing visual features extracted from a video sequence associated with the audio sequence and including current locations of face, lip movements and/or facial expressions of the speaker in a sequence of images in the video sequence;
• estimating a noise power density spectrum of statistically distributed background noise based on detection of the voice of the speaker;
• subtracting a discretized version of the estimated noise power density spectrum from the discrete signal spectrum of the digital audio sequence to obtain a difference signal;
• and
• calculating a corresponding discrete time-domain signal of the obtained difference signal by performing an Inverse Fast Fourier Transform (IFFT) to provide a recognized speech signal.

A telecommunication system, comprising:

• a video-enabled phone;
• a video-telephony based application running on the video-enabled phone;
• and
• a video camera built-in to the video-enabled phone and pointing at a face of a speaker participating in a video telephony session,
• wherein said video-telephony based application comprises:
• audio sequence detection means for detecting an analog audio sequence;
• audio feature extraction and analysis means for analyzing said analog audio sequence and extracting said audio features therefrom;
• video sequence detection means for detecting said video sequence;
• visual feature extraction and analysis means for analyzing the detected video sequence and extracting said visual features therefrom;
• noise reduction means for separating a speaker's voice from said background noise based on a combination of derived speech characteristics and outputting a speech activity indication signal comprising a combination of speech activity estimates supplied by said audio feature extraction and analysis means and said visual feature extraction and analysis means;
• and
• multi-channel acoustic echo cancellation means for performing a near-end speaker detection and double-talk detection algorithm based on the speech characteristics derived by said audio feature extraction and analysis means and said visual feature extraction and analysis means.