# US 12,087,297 B2 · Voice filtering other speakers from calls and audio messages

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,087,297 B2
- Document ID: US12087297B2
- Status: Granted
- Kind: B2
- Application number: 17/930,822
- Filed: 2022-09-09
- Published: 2023-01-05
- Granted: 2024-09-10
- Earliest claimed: 2020-10-30
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Matthew Sharifi; Victor Carbune
- Web page: https://patentgrep.com/patent/US12087297B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 21/028: using properties of sound source
- H04M 3/40: Applications of speech amplifiers
- H04M 3/4931: {Directory assistance systems}
- G10L 21/0208: Noise filtering
- G10L 21/0272: Voice signal separating
- G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10)
- G10L 25/87: Detection of discrete points within a voice signal
- G10L 2015/088: {Word spotting}
- H04M 3/568: {audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants (echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30)}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04M: TELEPHONIC COMMUNICATION (circuits for controlling other apparatus via a telephone cable and not involving telephone switching apparatus G08)

## Abstract

A method includes receiving a first instance of raw audio data corresponding to a voice-based command and receiving a second instance of the raw audio data corresponding to an utterance of audible contents for an audio-based communication spoken by a user. When a voice filtering recognition routine determines to activate voice filtering for at least the voice of the user, the method also includes obtaining a respective speaker embedding of the user and processing, using the respective speaker embedding, the second instance of the raw audio data to generate enhanced audio data for the audio-based communication that isolates the utterance of the audible contents spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user The method also includes executing.

## Claims

### Claim 1

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving raw audio data corresponding to an utterance of audible contents spoken by a user and captured by an assistant-enabled device, the raw audio data capturing one or more additional sounds that are not spoken by the user;
receiving, from an image capture device in communication with the data processing hardware, image data capturing the user while speaking the utterance of the audible contents;
extracting, from the image data, a facial image for the user;
extracting, from the raw audio data, audio features synchronized with lips of the user moving in the extracted facial image; and
processing, using the extracted audio features, the raw audio data to generate enhanced audio data that isolates the utterance of the audible contents spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise transmitting the enhanced audio data in an audio-based communication to a recipient device associated with a recipient of the audio-based communication, the enhanced audio data when received by the recipient device, causing the recipient device to audibly output the utterance of the audible contents spoken by the user.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, wherein the operations further comprise:
executing a voice filtering recognition routine to determine to activate voice filtering for a voice of the user in the audio-based communication based on the raw audio data,
wherein processing the raw audio data to generate the enhanced audio data is based on determining to activate voice filtering for the voice of the user.

### Claim 4 (depends on 3)

The computer-implemented method of claim 3, wherein executing the voice filtering recognition routine to determine to activate the voice filtering for the voice of the user comprises
identifying the recipient of the audio-based communication;
determining that the identified recipient of the audio-based communication comprises a particular recipient type indicating that activating the voice filtering for the voice of the user in the audio-based communication is appropriate; and
based on determining that the identified recipient of the audio-based communication comprises the particular recipient type, determining to activate voice filtering for at least the voice of the user.

### Claim 5 (depends on 4)

The computer-implemented method of claim 4, wherein the recipient type comprises a business.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
receiving initial audio data corresponding to a voice-based command for the assistant-enabled device to facilitate an audio-based communication between the user of the assistant-enabled device and a recipient, the voice-based command spoken by the user and captured by the assistant-enabled device;
processing using a speech recognizer, the initial audio data to generate a speech recognition result; and
performing semantic interpretation on the speech recognition result for the initial audio data to determine that the initial audio data comprises the voice-based command to facilitate the audio-based communication between the user and the recipient,
wherein the utterance of the audible contents spoken by the user is for the audio-based communication spoken.

### Claim 7 (depends on 6)

The computer-implemented method of claim 6, wherein the operations further comprise:
determining, based on the semantic interpretation performed on the speech recognition result for the initial audio data, that the voice-based command comprises an explicit instruction to activate voice filtering for the voice of the user,
wherein processing the raw audio data to generate the enhanced audio data is based on determining that the voice-based command comprises the explicit instruction to activate voice filtering.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the raw audio data comprises preamble audio and a hotword preceding the utterance of the audible contents.

### Claim 9 (depends on 8)

The computer-implemented method of claim 8, wherein the operations further comprise:
extracting audio features from the preamble audio and/or the hotword to determine a background noise level of an environment of the assistant-enabled device,
wherein processing the raw audio data to generate the enhanced audio data is based on the background noise level of the environment of the assistant-enabled device.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise displaying, in a graphical user interface (GUI) displayed on a screen in communication with the data processing hardware:
a graphical indicator indicating whether or not voice filtering is currently activated for the voice of the user; and
a control for activating/deactivating voice filtering for the voice of the user.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
receiving raw audio data corresponding to an utterance of audible contents spoken by a user and captured by an assistant-enabled device, the raw audio data capturing one or more additional sounds that are not spoken by the user;
receiving, from an image capture device in communication with the data processing hardware, image data capturing the user while speaking the utterance of the audible contents;
extracting, from the image data, a facial image for the user;
extracting, from the raw audio data, audio features synchronized with lips of the user moving in the extracted facial image; and
processing, using the extracted audio features, the raw audio data to generate enhanced audio data that isolates the utterance of the audible contents spoken by the user and excludes at least a portion of the one or more additional sounds that are not spoken by the user.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise transmitting the enhanced audio data in an audio-based communication to a recipient device associated with a recipient of the audio-based communication, the enhanced audio data when received by the recipient device, causing the recipient device to audibly output the utterance of the audible contents spoken by the user.

### Claim 13 (depends on 12)

The system of claim 12, wherein the operations further comprise:
executing a voice filtering recognition routine to determine to activate voice filtering for a voice of the user in the audio-based communication based on the raw audio data,
wherein processing the raw audio data to generate the enhanced audio data is based on determining to activate voice filtering for the voice of the user.

### Claim 14 (depends on 13)

The system of claim 13, wherein executing the voice filtering recognition routine to determine to activate the voice filtering for the voice of the user comprises
identifying the recipient of the audio-based communication;
determining that the identified recipient of the audio-based communication comprises a particular recipient type indicating that activating the voice filtering for the voice of the user in the audio-based communication is appropriate; and
based on determining that the identified recipient of the audio-based communication comprises the particular recipient type, determining to activate voice filtering for at least the voice of the user.

### Claim 15 (depends on 14)

The system of claim 14, wherein the recipient type comprises a business.

### Claim 16 (depends on 11)

The system of claim 11, wherein the operations further comprise:
receiving initial audio data corresponding to a voice-based command for the assistant-enabled device to facilitate an audio-based communication between the user of the assistant-enabled device and a recipient, the voice-based command spoken by the user and captured by the assistant-enabled device;
processing using a speech recognizer, the initial audio data to generate a speech recognition result; and
performing semantic interpretation on the speech recognition result for the initial audio data to determine that the initial audio data comprises the voice-based command to facilitate the audio-based communication between the user and the recipient,
wherein the utterance of the audible contents spoken by the user is for the audio-based communication spoken.

### Claim 17 (depends on 16)

The system of claim 16, wherein the operations further comprise:
determining, based on the semantic interpretation performed on the speech recognition result for the initial audio data, that the voice-based command comprises an explicit instruction to activate voice filtering for the voice of the user,
wherein processing the raw audio data to generate the enhanced audio data is based on determining that the voice-based command comprises the explicit instruction to activate voice filtering.

### Claim 18 (depends on 11)

The system of claim 11, wherein the raw audio data comprises preamble audio and a hotword preceding the utterance of the audible contents.

### Claim 19 (depends on 18)

The system of claim 18, wherein the operations further comprise:
extracting audio features from the preamble audio and/or the hotword to determine a background noise level of an environment of the assistant-enabled device,
wherein processing the raw audio data to generate the enhanced audio data is based on the background noise level of the environment of the assistant-enabled device.

### Claim 20 (depends on 11)

The system of claim 11, wherein the operations further comprise displaying, in a graphical user interface (GUI) displayed on a screen in communication with the data processing hardware:
a graphical indicator indicating whether or not voice filtering is currently activated for the voice of the user; and
a control for activating/deactivating voice filtering for the voice of the user.
