# US 12,217,761 B2 · Target speaker mode

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,217,761 B2
- Document ID: US12217761B2
- Status: Granted
- Kind: B2
- Application number: 17/515,480
- Filed: 2021-10-31
- Published: 2023-03-30
- Granted: 2025-02-04
- Earliest claimed: 2021-09-24
- Assignees: Zoom Video Communications, Inc.
- Applicants: Zoom Video Communications, Inc.
- Inventors: Yuhui Chen; Qiyong Liu; Zhengwei Wei; Yangbin Zeng
- Web page: https://patentgrep.com/patent/US12217761B2

## CPC

- G10L 17/08: Use of distortion metrics or a particular distance between probe pattern and reference templates
- G10L 21/0272: Voice signal separating
- G10L 17/04: Training, enrolment or model building
- G10L 25/21: the extracted parameters being power information
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction
- G10L 17/06: Decision making techniques; Pattern matching strategies
- G10L 25/30: using neural networks
- G10L 2021/02087: {the noise being separate speech, e.g. cocktail party}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media relate to a method for target speaker extraction. A target speaker extraction system receives an audio frame of an audio signal. A multi-speaker detection model analyzes the audio frame to determine whether the audio frame includes only a single-speaker or multiple speakers. When the audio frame includes only a single-speaker, the system inputs the audio frame to a target speaker VAD model to suppress speech in the audio frame from a non-target speaker based on comparing the audio frame to a voiceprint of a target speaker. When the audio frame includes multiple speakers, the system inputs the audio frame to a speech separation model to separate the voice of the target speaker from a voice mixture in the audio frame.

## Claims

### Claim 1

A computer-implemented method for target speaker extraction, comprising:
receiving, by a target speaker extraction system, an audio frame of an audio signal and a corresponding video, wherein the target speaker extraction system comprises a trained multi-speaker detection machine-learning (“ML”) model, a trained lip-movement-based (“LM-based”) target speaker voice activity detection (VAD) ML model, and a trained speech separation ML model;
responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a single speaker within the audio frame:
inputting, by the target speaker extraction system, the audio frame and the video to the trained LM-based target speaker VAD ML model; and
suppressing, by the trained LM-based target speaker VAD ML model of the target speaker extraction system and based on the video, speech in the audio frame from a non-target speaker, wherein suppressing the speech in the audio from a non-target speaker comprises comparing the audio frame to a voiceprint of a target speaker; and
responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a plurality of speakers within the audio frame:
inputting, by the target speaker extraction system, the audio frame to the trained speech separation ML model; and
separating, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker from a voice mixture in the audio frame.

### Claim 2 (depends on 1)

The method of claim 1, wherein suppressing, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, speech in the audio frame from the non-target speaker further comprises:
determining, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, a suppression ratio; and
suppressing, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the speech in the audio frame from the non-target speaker based on the suppression ratio.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
generating, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, a voiceprint of the non-target speaker;
comparing, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the voiceprint of the non-target speaker to the voiceprint of the target speaker to determine a similarity score; and
determining, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the suppression ratio based on the similarity score.

### Claim 4 (depends on 1)

The method of claim 1, wherein:
the method further comprises:
receiving, by the target speaker extraction system, a voice recording from a target speaker prior to a video conference; and
generating, by the target speaker extraction system, the voiceprint of the target speaker from the voice recording prior to the video conference; and
separating, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker from the voice mixture in the audio frame further comprises:
extracting, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker during the video conference based on the voiceprint of the target speaker.

### Claim 5 (depends on 1)

The method of claim 1, further comprising:
determining, by the target speaker extraction system, an energy of the audio signal, wherein the audio signal is received during a video conference;
determining, by the target speaker extraction system, speech by the target speaker within the audio frame based on the energy of the audio signal; and
generating, by the target speaker extraction system, a voiceprint of the target speaker from the audio signal.

### Claim 6 (depends on 5)

The method of claim 5, wherein determining, by the target speaker extraction system, the speech by the target speaker within the audio frame based on the energy of the audio signal comprises determining that the energy exceeds a threshold.

### Claim 7 (depends on 1)

The method of claim 1, wherein the target speaker extraction system comprises a trained voiceprint extraction ML model, and the method further comprises:
generating, by the trained voiceprint extraction ML model of the target speaker extraction system, a voiceprint of the target speaker; and
sharing, by the trained voiceprint extraction ML model, one or more weights associated with the voiceprint with the trained speech separation ML model.

### Claim 8

A non-transitory computer readable medium comprising processor-executable instructions configured to cause one or more processors to:
receive, by a target speaker extraction system, an audio frame of an audio signal and a corresponding video, wherein the target speaker extraction system comprises a trained multi-speaker detection machine-learning (“ML”) model, a trained lip-movement-based (“LM-based”) target speaker voice activity detection (VAD) ML model, and a trained speech separation ML model;
responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a single speaker within the audio frame:
input, by the target speaker extraction system, the audio frame and the video to the trained LM-based target speaker VAD ML model; and
suppress, by the trained LM-based target speaker VAD ML model of the target speaker extraction system and based on the video, speech in the audio frame from a non-target speaker, wherein suppressing the speech in the audio from a non-target speaker comprises comparing the audio frame to a voiceprint of a target speaker; and
responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a plurality of speakers within the audio frame:
input, by the target speaker extraction system, the audio frame to the trained speech separation ML model; and
separate, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker from a voice mixture in the audio frame.

### Claim 9 (depends on 8)

The non-transitory computer readable medium of claim 8, further comprising processor-executable instructions configured to cause the one or more processors to:
determine, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, a suppression ratio; and
suppress, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the speech in the audio frame from the non-target speaker based on the suppression ratio.

### Claim 10 (depends on 9)

The non-transitory computer readable medium of claim 9, further comprising processor-executable instructions configured to cause the one or more processors to:
generate, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, a voiceprint of the non-target speaker;
compare, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the voiceprint of the non-target speaker to the voiceprint of the target speaker to determine a similarity score; and
determine, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the suppression ratio based on the similarity score.

### Claim 11 (depends on 8)

The non-transitory computer readable medium of claim 8, further comprising processor-executable instructions configured to cause the one or more processors to:
receive, by the target speaker extraction system, a voice recording from a target speaker prior to a video conference; and
generate, by the target speaker extraction system, the voiceprint of the target speaker from the voice recording prior to the video conference; and
extract, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker during the video conference based on the voiceprint of the target speaker.

### Claim 12 (depends on 8)

The non-transitory computer readable medium of claim 8, further comprising processor-executable instructions configured to cause the one or more processors to:
determining, by the target speaker extraction system, an energy of the audio signal, wherein the audio signal is received during a video conference;
determining, by the target speaker extraction system, speech by the target speaker within the audio frame based on the energy of the audio signal; and
generating, by the target speaker extraction system, a voiceprint of the target speaker from the audio signal.

### Claim 13 (depends on 12)

The non-transitory computer readable medium of claim 12, wherein the processor-executable instructions configured to cause the one or more processors to determine, by the target speaker extraction system, the speech by the target speaker within the audio frame based on the energy of the audio signal further comprise further comprise processor-executable instructions configured to cause the one or more processors to determine that the energy exceeds a threshold.

### Claim 14 (depends on 8)

The non-transitory computer readable medium of claim 8, wherein the target speaker extraction system comprises a trained voiceprint extraction ML model, and further comprising processor-executable instructions configured to cause the one or more processors to:
generate, by the trained voiceprint extraction ML model of the target speaker extraction system, a voiceprint of the target speaker; and
share, by the trained voiceprint extraction ML model, one or more weights associated with the voiceprint with the trained speech separation ML model.

### Claim 15

A target speaker extraction system comprising:
a non-transitory computer-readable medium; and
one or more processors communicatively coupled to the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
receive, by a target speaker extraction system, an audio frame of an audio signal and a corresponding video, wherein the target speaker extraction system comprises a trained multi-speaker detection machine-learning (“ML”) model, a trained lip-movement-based (“LM-based”) target speaker voice activity detection (VAD) ML model, and a trained speech separation ML model;
responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a single speaker within the audio frame:
input, by the target speaker extraction system, the audio frame and the video to the trained LM-based target speaker VAD ML model; and
suppress, by the trained LM-based target speaker VAD ML model of the target speaker extraction system and based on the video, speech in the audio frame from a non-target speaker, wherein suppressing the speech in the audio from a non-target speaker comprises comparing the audio frame to a voiceprint of a target speaker; and
responsive to determining, by the trained multi-speaker detection ML model of the target speaker extraction system, a plurality of speakers within the audio frame:
input, by the target speaker extraction system, the audio frame to the trained speech separation ML model; and
separate, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker from a voice mixture in the audio frame.

### Claim 16 (depends on 15)

The system of claim 15, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to-further configured to
determine, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, a suppression ratio; and
suppress, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the speech in the audio frame from the non-target speaker based on the suppression ratio.

### Claim 17 (depends on 16)

The system of claim 16, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:
generate, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, a voiceprint of the non-target speaker;
compare, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the voiceprint of the non-target speaker to the voiceprint of the target speaker to determine a similarity score; and
determine, by the trained LM-based target speaker VAD ML model of the target speaker extraction system, the suppression ratio based on the similarity score.

### Claim 18 (depends on 15)

The system of claim 15, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to:
receive, by the target speaker extraction system, a voice recording from a target speaker prior to a video conference; and
generate, by the target speaker extraction system, the voiceprint of the target speaker from the voice recording prior to the video conference; and
extract, by the trained speech separation ML model of the target speaker extraction system, the voice of the target speaker during the video conference based on the voiceprint of the target speaker.

### Claim 19 (depends on 15)

The system of claim 15, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to further configured to:
determine, by the target speaker extraction system, an energy of the audio signal, wherein the audio signal is received during a video conference;
determine, by the target speaker extraction system, speech by the target speaker within the audio frame based on the energy of the audio signal; and
generate, by the target speaker extraction system, a voiceprint of the target speaker from the audio signal.

### Claim 20 (depends on 19)

The system of claim 19, wherein the one or more processors are configured to execute further processor-executable instructions stored in the non-transitory computer-readable medium to determine, by the target speaker extraction system, the speech by the target speaker within the audio frame based on the energy of the audio signal exceeding a threshold.