# US 2021/0241780 A1 · Method And System For Speech Enhancement

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0241780 A1
- Document ID: US20210241780A1
- Status: Published application
- Kind: A1
- Application number: 16/779,145
- Filed: 2020-01-31
- Published: 2021-08-05
- Earliest claimed: 2020-01-31
- Assignees: Nuance Communications, Inc.
- Applicants: Nuance Communications, Inc.
- Inventors: Carl Benjamin Quillen
- Web page: https://patentgrep.com/patent/US20210241780A1

## CPC

- G10L 21/0208: Noise filtering
- G10L 21/0216: characterised by the method used for estimating noise
- G06N 3/02: Neural networks
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 21/0324: Details of processing therefor
- G10L 25/30: using neural networks
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Embodiments improve speech data quality through training a neural network for de-noising audio enhancement. One such embodiment creates simulated noisy speech data from high quality speech data. In turn, training, e.g., deep normalizing flow training, is performed on a neural network using the high quality speech data and the simulated noisy speech data to train the neural network to create de-noised speech data given noisy speech data. Performing the training includes minimizing errors in the neural network according to at least one of (i) a decoding error of an Automatic Speech Recognition (ASR) system processing current de-noised speech data results generated by the neural network during the training and (ii) spectral distance between the high quality speech data and the current de-noised speech data results generated by the neural network during the training.

## Claims

### Claim 1

A method of training a neural network for de-noising audio enhancement, the method comprising:
creating simulated noisy speech data from high quality speech data; and
performing training on a neural network using the high quality speech data and the simulated noisy speech data to train the neural network to create de-noised speech data given noisy speech data, wherein performing the training includes minimizing errors in the neural network according to at least one of:
a decoding error of an Automatic Speech Recognition (ASR) system processing current de-noised speech data results that are generated by the neural network during the training; and
spectral distance between the high quality speech data and the current de-noised speech data results that are generated by the neural network during the training.

### Claim 2 (depends on 1)

The method of claim 1 further comprising:
generating the current de-noised speech data results during the training by processing at least a portion of the simulated noisy speech data with the neural network.

### Claim 3 (depends on 2)

The method of claim 2 further comprising:
determining the decoding error during the training by comparing (1) speech recognition results generated by the ASR system processing the current de-noised speech data results and (2) a transcript of at least a portion of the high quality speech data upon which the at least a portion of the simulated noisy speech data was created.

### Claim 4 (depends on 1)

The method of claim 1 further comprising:
collecting the high quality speech data in a low noise environment.

### Claim 5 (depends on 1)

The method of claim 1 further comprising:
creating the simulated noisy speech data by adding reverberation to the high quality speech data using convolution.

### Claim 6 (depends on 5)

The method of claim 5 further comprising:
adding the reverberation using convolution by accessing a database comprising at least one of: measured impulse responses from a reverberant environment and synthetically generated impulse responses.

### Claim 7 (depends on 1)

The method of claim 1 further comprising:
collecting data from an environment in which the ASR system is to be deployed; and
creating the simulated noisy speech data in accordance with the data collected from the environment.

### Claim 8 (depends on 1)

The method of claim 1 further comprising:
performing the training by training the neural network to learn a maximum-likely encryption of the high quality speech data given the simulated noisy speech data.

### Claim 9 (depends on 1)

The method of claim 1 wherein minimizing the errors in the neural network includes:
adjusting one or more weights of the neural network.

### Claim 10 (depends on 1)

The method of claim 1 further comprising:
after the training, processing noisy speech data using the trained neural network to determine enhanced speech data.

### Claim 11 (depends on 1)

The method of claim 1 wherein the training is deep normalizing flow training.

### Claim 12

A computer system for training a neural network for de-noising audio enhancement, the computer system comprising:
a processor; and
a memory with computer code instructions stored thereon, the processor and the memory, with the computer code instructions being configured to cause the system to:
create simulated noisy speech data from high quality speech data; and
perform training on a neural network using the high quality speech data and the simulated noisy speech data to train the neural network to create de-noised speech data given noisy speech data, wherein performing the training includes minimizing errors in the neural network according to at least one of:
a decoding error of an Automatic Speech Recognition (ASR) system processing current de-noised speech data results that are generated by the neural network during the training; and
spectral distance between the high quality speech data and the current de-noised speech data results that are generated by the neural network during the training.

### Claim 13 (depends on 12)

The system of claim 12 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
generate the current de-noised speech data results during the training by processing at least a portion of the simulated noisy speech data with the neural network.

### Claim 14 (depends on 13)

The system of claim 13 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
determine the decoding error during the training by comparing (1) speech recognition results generated by the ASR system processing the current de-noised speech data results and (2) a transcript of at least a portion of the high quality speech data upon which the at least a portion of the simulated noisy speech data was created.

### Claim 15 (depends on 12)

The system of claim 12 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
collect the high quality speech data in a low noise environment.

### Claim 16 (depends on 12)

The system of claim 12 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
create the simulated noisy speech data by adding reverberation to the high quality speech data using convolution.

### Claim 17 (depends on 16)

The system of claim 16 wherein, in adding the reverberation using convolution, the processor and the memory, with the computer code instructions, are further configured to cause the system to:
access a database comprising at least one of: measured impulse responses from a reverberant environment and synthetically generated impulse responses.

### Claim 18 (depends on 12)

The system of claim 12 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
collect data from an environment in which the ASR system is to be deployed; and
create the simulated noisy speech data in accordance with the data collected from the environment.

### Claim 19 (depends on 12)

The system of claim 12 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
perform the training by training the neural network to learn a maximum-likely encryption of the high quality speech data given the simulated noisy speech data.

### Claim 20 (depends on 12)

The system of claim 12 wherein the processor and the memory, with the computer code instructions, are further configured to cause the system to:
after the training, process noisy speech data using the trained neural network to determine enhanced speech data.

### Claim 21 (depends on 12)

The system of claim 12 wherein, in performing the training on a neural network, the processor and the memory, with the computer code instructions, are further configured to cause the system to:
perform deep normalizing flow training.

### Claim 22

A non-transitory computer program product for training a neural network for de-noising audio enhancement, the computer program product comprising:
one or more computer-readable storage devices and program instructions stored on at least one of the one or more storage devices, the program instructions, when loaded and executed by a processor, cause an apparatus associated with the processor to:
create simulated noisy speech data from high quality speech data; and
perform training on a neural network using the high quality speech data and the simulated noisy speech data to train the neural network to create de-noised speech data given noisy speech data, wherein performing the training includes minimizing errors in the neural network according to at least one of:
a decoding error of an Automatic Speech Recognition (ASR) system processing current de-noised speech data results that are generated by the neural network during the training; and
spectral distance between the high quality speech data and the current de-noised speech data results that are generated by the neural network during the training.
