Method and system for enhancing a speech signal of a human speaker in a video using visual information

A system for enhancing a speech signal of a target human speaker, the system comprising:

• a computer memory configured to obtain a video, wherein said video comprises a sequence of images showing a face or parts of a face of the target human speaker, and an original soundtrack corresponding with said video;
• and
• a computer processor configured to:
• represent said original soundtrack by a two-dimensional (2D) discrete time-frequency (DTF) audio transform;
• generate a 2D time-frequency filter, having same dimensions as said 2D DTF audio transform, by analyzing said video;
• obtain a filtered DTF audio transform by point-wise multiplying the 2D DTF audio transform by said 2D time-frequency filter;
• and
• generate an enhanced speech signal based on said filtered DTF audio transform, wherein said enhanced speech signal exhibits a removal, from the original soundtrack, of sounds that are unrelated to the speech of said target human speaker.

A non-transitory computer readable medium for enhancing a speech signal of a target human speaker, the non-transitory computer readable medium comprising a set of instructions that when executed cause at least one computer processor to:

• represent said original soundtrack by a two-dimensional (2D) discrete time-frequency (DTF) audio transform;
• generate a 2D time-frequency filter, having same dimensions as said 2D DTF audio transform, by analyzing said video;
• obtain a filtered DTF audio transform by point-wise multiplying the 2D DTF audio transform by said 2D time-frequency filter;
• and generate an enhanced speech signal based on said filtered DTF audio transform, wherein said enhanced speech signal exhibits a removal, from the original soundtrack, of sounds that are unrelated to the speech of said target human speaker.