Methods and systems of noise aware audio visual speech denoising

A system for de-noising an audio visual speech, the system comprising:

• a modeling engine configured to model a noise in the audio visual speech using a noisy speech from audio data associated with the audio visual speech to generate a reconstructed noise signal, the audio data comprising audio features;
• an estimation engine configured to estimate the reconstructed noise signal in the audio visual speech using an audio signal and a plurality of visual frames associated with the audio visual speech;
• a partitioning engine configured to partition the reconstructed noise signal into a plurality of windows in a time domain and calculate an energy associated with each window amongst the plurality of windows;
• a noise strength estimation engine configured to estimate a noise strength in each window by performing a soft max operation over time domain of the reconstructed noise signal, wherein the noise strength in each window is used to obtain one or more refined audio features based on an amount of noise strength in the window;
• and
• a fusing engine configured to fuse the one or more refined audio features along with one or more visual features associated with the audio visual speech using the noise strength to generate an output, wherein the output is passed through a decoder to obtain a de-noised audio visual speech.

A non-transitory machine-readable storage medium including instructions that, when executed by a machine, cause the machine to perform operations comprising:

• modeling, by a modeling engine, a noise in an audio visual speech using a noisy speech from audio data associated with the audio visual speech to generate a reconstructed noise signal, the audio data comprising audio features;
• estimating, by an estimation engine, the reconstructed noise signal in the audio visual speech using an audio signal and a plurality of visual frames associated with the audio visual speech;
• partitioning, by a partitioning engine, the reconstructed noise signal into a plurality of windows in a time domain and calculating an energy associated with each window amongst the plurality of windows;
• estimating, by a noise strength estimation engine, a noise strength in each window by performing a soft max operation over time domain of the reconstructed noise signal, wherein the noise strength in each window is used to obtain one or more refined audio features based on an amount of noise strength in the window;
• and
• fusing, by a fusing engine, the one or more refined audio features along with one or more visual features associated with the audio visual speech using the noise strength to generate an output, wherein the output is passed through a decoder to obtain a de-noised audio visual speech.