Single-channel and multi-channel source separation enhanced by lip motion

A method comprising:

• extracting a lip motion vector (“LMV”) from facial features of a target speaker facial image captured over a period of interest;
• augmenting the LMV by voice activity detection (“VAD”) output;
• and
• separating, based on at least the motion vectors, audio corresponding to a constituent source from a multi-channel audio signal captured over the period of interest;
• wherein separating audio from the audio signal is performed by a blind source separation (“BSS”) learning model outputting a demixing matrix.

A system comprising:

• one or more processors;
• and
• memory communicatively coupled to the one or more processors, the memory storing computer-executable modules executable by the one or more processors that, when executed by the one or more processors, perform associated operations, the computer-executable modules comprising:
• a facial feature extracting module, further comprising:
• a lip motion vector computing submodule configured to extract a lip motion vector (“LMV”) from facial features of a target speaker facial image captured over a period of interest and augment the LMV by voice activity detection (“VAD”) output;
• and
• a source separating module configured to separate, based on at least the motion vectors, audio corresponding to a constituent source from a multi-channel audio signal captured over the period of interest;
• wherein the source separating module is configured to separate audio from the audio signal is performed by a blind source separation (“BSS”) learning model outputting a demixing matrix.

A computer-readable storage medium storing computer-readable instructions executable by one or more processors, that when executed by the one or more processors, cause the one or more processors to perform operations comprising:

• extracting a lip motion vector (“LMV”) from facial features of a target speaker facial image captured over a period of interest;
• augmenting the LMV by voice activity detection (“VAD”) output;
• and
• separating, based on at least the motion vectors, audio corresponding to a constituent source from a multi-channel audio signal captured over the period of interest;
• wherein separating audio from the audio signal is performed by a blind source separation (“BSS”) learning model outputting a demixing matrix.