Target speaker mode

A non-transitory computer readable medium comprising processor-executable instructions configured to cause one or more processors to:

• receive, by a target speaker extraction system, an audio frame of an audio signal and a corresponding video, wherein the target speaker extraction system comprises a trained multi-speaker detection machine-learning (“ML”) model, a trained lip-movement-based (“LM-based”) target speaker voice activity detection (VAD) ML model, and a trained speech separation ML model;
• responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a single speaker within the audio frame:
• input, by the target speaker extraction system, the audio frame and the video to the trained LM-based target speaker VAD ML model;
• and
• suppress, by the trained LM-based target speaker VAD ML model of the target speaker extraction system and based on the video, speech in the audio frame from a non-target speaker, wherein suppressing the speech in the audio from a non-target speaker comprises comparing the audio frame to a voiceprint of a target speaker;
• and
• responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a plurality of speakers within the audio frame:
• input, by the target speaker extraction system, the audio frame to the trained speech separation ML model;
• and
• separate, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker from a voice mixture in the audio frame.

A target speaker extraction system comprising:

• a non-transitory computer-readable medium;
• and
• one or more processors communicatively coupled to the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
• receive, by a target speaker extraction system, an audio frame of an audio signal and a corresponding video, wherein the target speaker extraction system comprises a trained multi-speaker detection machine-learning (“ML”) model, a trained lip-movement-based (“LM-based”) target speaker voice activity detection (VAD) ML model, and a trained speech separation ML model;
• responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a single speaker within the audio frame:
• input, by the target speaker extraction system, the audio frame and the video to the trained LM-based target speaker VAD ML model;
• and
• suppress, by the trained LM-based target speaker VAD ML model of the target speaker extraction system and based on the video, speech in the audio frame from a non-target speaker, wherein suppressing the speech in the audio from a non-target speaker comprises comparing the audio frame to a voiceprint of a target speaker;
• and
• responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a plurality of speakers within the audio frame:
• input, by the target speaker extraction system, the audio frame to the trained speech separation ML model;
• and
• separate, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker from a voice mixture in the audio frame.