# US 2021/0343306 A1 · Mixed adaptive and fixed coefficient neural networks for speech enhancement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343306 A1
- Document ID: US20210343306A1
- Status: Published application
- Kind: A1
- Application number: 17/243,434
- Filed: 2021-04-28
- Published: 2021-11-04
- Earliest claimed: 2020-04-29
- Assignees: QUALCOMM Incorporated
- Applicants: QUALCOMM Incorporated
- Inventors: Erik VISSER; Vahid MONTAZERI; Shuhua ZHANG; Lae-Hoon KIM
- Web page: https://patentgrep.com/patent/US20210343306A1

## CPC

- G10L 21/0208: Noise filtering
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0475: Generative networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 25/30: using neural networks
- G10L 2021/02082: {the noise being echo, reverberation of the speech}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems, methods and computer-readable media are provided for speech enhancement using a hybrid neural network. An example process can include receiving, by a first neural network portion of the hybrid neural network, audio data and reference data, the audio data including speech data, noise data, and echo data; filtering, by the first neural network portion, a portion of the audio data based on adapted coefficients of the first neural network portion, the portion of the audio data including the noise data and/or echo data; based on the filtering, generating, by the first neural network portion, filtered audio data including the speech data and an unfiltered portion of the noise data and/or echo data; and based on the filtered audio data and the reference data, extracting, by a second neural network portion of the hybrid neural network, the speech data from the filtered audio data.

## Claims

### Claim 1

An apparatus comprising:
memory; and
one or more processors coupled to the memory, the one or more processors being configured to:
receive, by a first neural network portion of a hybrid neural network system, an input comprising audio data and reference data, the audio data comprising speech data, noise data, and echo data;
filter, by the first neural network portion, a portion of the audio data based on adapted coefficients of the first neural network portion, the portion of the audio data comprising at least one of the noise data and/or the echo data, the adapted coefficients comprising coefficients adjusted based on the input and/or an output of the first neural network portion;
generate, by the first neural network portion based on the filtering of the portion of the audio data, filtered audio data comprising the speech data and an unfiltered portion of at least one of the noise data and/or the echo data; and
extract, by a second neural network portion of the hybrid neural network system based on the filtered audio data and the reference data, the speech data from the filtered audio data.

### Claim 2 (depends on 1)

The apparatus of claim 1, wherein the echo data comprises non-linear echo data and linear echo data, and wherein the portion filtered by the first neural network portion comprises the linear echo data.

### Claim 3 (depends on 2)

The apparatus of claim 2, wherein the reference data comprises far end echo data, wherein the adapted coefficients of the first neural network portion are linear coefficients that are adapted based on an approximation of the linear echo data, the approximation of the linear echo data being based on the far end echo data.

### Claim 4 (depends on 2)

The apparatus of claim 2, wherein the unfiltered portion of at least one of the noise data and/or the echo data comprises the non-linear echo data, and wherein the filtered audio data comprises the speech data, the noise data and the non-linear echo data.

### Claim 5 (depends on 1)

The apparatus of claim 1, wherein the first neural network portion comprises an adaptive neural network portion, and wherein the second neural network portion comprises a fixed coefficient neural network portion.

### Claim 6 (depends on 1)

The apparatus of claim 1, wherein the one or more processors are configured to:
filter, based on the extracted speech data, an additional portion of at least one of the noise data and/or the echo data.

### Claim 7 (depends on 6)

The apparatus of claim 6, wherein, to filter the additional portion of at least one of the noise data and/or the echo data, the one or more processors are configured to:
provide, by the second neural network portion to the first neural network portion, feedback comprising the extracted speech data.

### Claim 8 (depends on 6)

The apparatus of claim 6, wherein, to filter the additional portion of at least one of the noise data and/or the echo data, the one or more processors are configured to:
provide the extracted speech data and the input to a third neural network portion of the hybrid neural network system;
filter, by the third neural network portion based on additional adapted coefficients of the third neural network portion, a different portion of at least one of the noise data and/or the echo data; and
generate, by the third neural network portion, different filtered audio data comprising the speech data.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the one or more processors are configured to:
provide, to a fourth neural network portion of the hybrid neural network system, the different filtered audio data and the reference data; and
output, by the fourth neural network portion, different extracted speech data based on the different filtered audio data and the reference data.

### Claim 10 (depends on 1)

The apparatus of claim 1, wherein the first neural network portion receives the audio data and adjusts the coefficients in real time to generate the adapted coefficients.

### Claim 11 (depends on 1)

The apparatus of claim 1, wherein to filter the portion of the audio data, the one or more processors are configured to:
iteratively approximate the adapted coefficients, wherein a first iteration implements initial coefficient values for the coefficients, and wherein the adapted coefficients are generated at one or more additional iterations.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein the one or more processors are configured to:
receive second audio data; and
re-approximate, by the first neural network portion based on the second audio data, the adapted coefficients, the re-approximating starting with the adapted coefficients as the initial coefficient values.

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein the one or more processors are configured to:
receive second audio data; and
generate, by the first neural network portion based on the second audio data, second filtered audio data based on the adapted coefficients.

### Claim 14 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises a mobile device.

### Claim 15 (depends on 1)

The apparatus of claim 1, wherein the apparatus comprises at least one of a microphone and/or a speaker device.

### Claim 16

A method comprising:
receiving, by a first neural network portion of a hybrid neural network system, an input comprising audio data and reference data, the audio data comprising speech data, noise data, and echo data;
filtering, by the first neural network portion, a portion of the audio data based on adapted coefficients of the first neural network portion, the portion of the audio data comprising at least one of the noise data and/or the echo data, the adapted coefficients comprising coefficients adjusted based on the input and/or an output of the first neural network portion;
generating, by the first neural network portion based on the filtering of the portion of the audio data, filtered audio data comprising the speech data and an unfiltered portion of at least one of the noise data and/or the echo data; and
extracting, by a second neural network portion of the hybrid neural network system based on the filtered audio data and the reference data, the speech data from the filtered audio data.

### Claim 17 (depends on 16)

The method of claim 16, wherein the echo data comprises non-linear echo data and linear echo data, and wherein the portion filtered by the first neural network portion comprises the linear echo data.

### Claim 18 (depends on 17)

The method of claim 17, wherein the reference data comprises far end echo data, wherein the adapted coefficients of the first neural network portion are linear coefficients that are adapted based on an approximation of the linear echo data, the approximation of the linear echo data being based on the far end echo data.

### Claim 19 (depends on 17)

The method of claim 17, wherein the unfiltered portion of at least one of the noise data and/or the echo data comprises the non-linear echo data, and wherein the filtered audio data comprises the speech data, the noise data and the non-linear echo data.

### Claim 20 (depends on 16)

The method of claim 16, wherein the first neural network portion comprises an adaptive neural network portion, and wherein the second neural network portion comprises a fixed coefficient neural network portion.

### Claim 21 (depends on 16)

The method of claim 16, further comprising:
filtering, based on the extracted speech data, an additional portion of at least one of the noise data and/or the echo data.

### Claim 22 (depends on 21)

The method of claim 21, wherein filtering the additional portion of at least one of the noise data and/or the echo data comprises:
providing, by the second neural network portion to the first neural network portion, feedback comprising the extracted speech data.

### Claim 23 (depends on 21)

The method of claim 21, wherein filtering the additional portion of at least one of the noise data and/or the echo data comprises:
providing the extracted speech data and the input to a third neural network portion of the hybrid neural network system;
filtering, by the third neural network portion based on additional adapted coefficients of the third neural network portion, a different portion of at least one of the noise data and/or the echo data; and
generating, by the third neural network portion, different filtered audio data comprising the speech data.

### Claim 24 (depends on 23)

The method of claim 23, further comprising:
providing, to a fourth neural network portion of the hybrid neural network system, the different filtered audio data and the reference data; and
outputting, by the fourth neural network portion, different extracted speech data based on the different filtered audio data and the reference data.

### Claim 25 (depends on 16)

The method of claim 16, wherein the first neural network portion receives the audio data and adjusts the coefficients in real time to generate the adapted coefficients.

### Claim 26 (depends on 16)

The method of claim 16, wherein filtering the portion of the audio data comprises:
iteratively approximating the adapted coefficients, wherein a first iteration implements initial coefficient values for the coefficients, and wherein the adapted coefficients are generated at one or more additional iterations.

### Claim 27 (depends on 26)

The method of claim 26, further comprising:
receiving second audio data; and
re-approximating, by the first neural network portion based on the second audio data, the adapted coefficients, the re-approximating starting with the adapted coefficients as the initial coefficient values.

### Claim 28 (depends on 26)

The method of claim 26, further comprising:
receiving second audio data; and
generating, by the first neural network portion based on the second audio data, second filtered audio data based on the adapted coefficients.

### Claim 29

A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to:
receive, by a first neural network portion of a hybrid neural network system, an input comprising audio data and reference data, the audio data comprising speech data, noise data, and echo data;
filter, by the first neural network portion, a portion of the audio data based on adapted coefficients of the first neural network portion, the portion of the audio data comprising at least one of the noise data and/or the echo data, the adapted coefficients comprising coefficients adjusted based on the input and/or an output of the first neural network portion;
generate, by the first neural network portion based on the filtering of the portion of the audio data, filtered audio data comprising the speech data and an unfiltered portion of at least one of the noise data and/or the echo data; and
extract, by a second neural network portion of the hybrid neural network system based on the filtered audio data and the reference data, the speech data from the filtered audio data.

### Claim 30 (depends on 29)

The non-transitory computer-readable medium of claim 29, wherein the echo data comprises non-linear echo data and linear echo data, and wherein the portion filtered by the first neural network portion comprises the linear echo data.
