# US 2019/0318755 A1 · Systems, methods, and computer-readable media for improved real-time audio processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0318755 A1
- Document ID: US20190318755A1
- Status: Published application
- Kind: A1
- Application number: 15/952,353
- Filed: 2018-04-13
- Published: 2019-10-17
- Earliest claimed: 2018-04-13
- Assignees: Microsoft Technology Licensing, LLC
- Applicants: Microsoft Technology Licensing, LLC
- Inventors: Ivan Jelev TASHEV; Shuayb M ZARAR; Yan-Hui TU; Chin-Hui LEE; Han ZHAO
- Web page: https://patentgrep.com/patent/US20190318755A1

## CPC

- G10L 21/0232: Processing in the frequency domain
- G10L 21/0364: for improving intelligibility
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/09: Supervised learning
- G10L 25/30: using neural networks
- G10L 25/84: for discriminating voice from noise
- G10L 15/16: using artificial neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems, methods, and computer-readable storage devices are disclosed for improved real-time audio processing. One method including: receiving audio data including a plurality of frames having a plurality of frequency bins; calculating, for each frequency bin, an approximate speech signal estimation based on the plurality of frames; calculating, for each approximate speech signal estimation, a clean speech estimation and at least one additional target including an ideal ratio mask using a trained neural network model; and calculating, for each frequency bin, a final clean speech estimation using the calculated at least one additional target including the calculated ideal ratio mask and the calculated clean speech estimation.

## Claims

### Claim 1

A computer-implemented method for improved real-time audio processing, the method comprising:
receiving audio data including a plurality of frames having a plurality of frequency bins;
calculating, for each frequency bin, an approximate speech signal estimation based on the plurality of frames;
calculating, for each approximate speech signal estimation, a clean speech estimation and at least one additional target including an ideal ratio mask using a trained neural network model; and
calculating, for each frequency bin, a final clean speech estimation using the calculated at least one additional target including the calculated ideal ratio mask and the calculated clean speech estimation.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
calculating, for each frequency bin, a prior signal-to-noise ratio and a posterior signal-to-noise ratio based on the plurality of frames; and
calculating, for each frequency bin, a suppression gain based on the calculated prior signal-to-noise ratio and the calculated posterior signal-to-noise ratio,
wherein calculating, for each frequency bin, the approximate speech signal estimation based on the plurality of frames includes:
calculating, for each frequency bin, the approximate speech signal estimation based on the calculated suppression gain and the ideal ratio mask.

### Claim 3 (depends on 2)

The method of claim 2, wherein the ideal ratio mask is calculated based on a proportion of powers of a clean speech to a noisy speech in a corresponding time-frequency bin.

### Claim 4 (depends on 1)

The method of claim 1, wherein the trained neural network model comprises a recurrent neural-network based on at least one long short-term memory layer.

### Claim 5 (depends on 1)

The method according to claim 1, further comprising:
constructing a neural network model, including a plurality of neurons, configured to output the clean speech estimation and the ideal ratio mask of audio data, the plurality of neurons arranged in a plurality of layers, including at least one hidden layer, and being connected by a plurality of connections;
receiving a training data set, the training data set including audio data;
training the neural network model using the training data set; and
outputting the trained neural network model configured to output the clean speech estimation and the ideal ratio mask of audio data.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the at least one hidden layer of the neural network model comprises at least one long short-term memory layer.

### Claim 7 (depends on 5)

The method according to claim 5, wherein training the neural network model using the training data set comprises:
training the neural network model using one or both of (i) a log-power spectra of a training data set and (ii) a linear-power spectra of a training data set as input features and one or both of (i) a clean log-power spectra and (ii) a clean linear-power spectra and reference ideal ratio mask.

### Claim 8 (depends on 5)

The method according to claim 5, wherein the training data set comprises a synthetic data set with separately known clean speech and noise signals for training.

### Claim 9 (depends on 7)

The method according to claim 7, wherein training the neural network model further comprises:
minimizing a mean square error between one of (i) both the log-power spectra and the clean log-power spectra and the calculated ideal ratio mask and the reference ideal ratio mask, or (ii) both the linear-power spectra and the clean linear-power spectra and the calculated ideal ratio mask and the reference ideal ratio mask.

### Claim 10 (depends on 1)

The method according to claim 1, further comprising:
receiving a test data set, the test data set including audio data with unseen noise; and
evaluating the trained deep neural network using the received test data set.

### Claim 11 (depends on 1)

The method according to claim 1, wherein the trained neural network model comprises a convolutional-recurrent neural network model.

### Claim 12 (depends on 11)

The method according to claim 11, wherein the convolutional-recurrent neural network model includes a convolutional component, at least one bidirectional recurrent component, and a fully-connected layer.

### Claim 13 (depends on 12)

The method according to claim 12, wherein the convolutional component exploits local patterns in a spectrogram in both a spatial domain and a temporal domain,
wherein the at least one bidirectional recurrent component models dynamic correlations between consecutive frames, and
wherein the fully-connected layer predicts clean spectrograms.

### Claim 14 (depends on 12)

The method according to claim 12, wherein the convolutional component convolves a noisy spectrogram with kernels to form feature maps, and the convolutional component concatenates the formed feature maps to form a 2D feature map.

### Claim 15 (depends on 14)

The method according to claim 14, wherein the bidirectional recurrent component transforms the 2D feature map by a bidirectional recurrent neural network along a time dimension.

### Claim 16 (depends on 12)

The method according to claim 12, wherein the bidirectional recurrent component automatically models dynamic correlations between adjacent frames in a noisy spectrogram by using a bidirectional recurrent neural network that includes recurrent connections in both directions.

### Claim 17

A system for improved real-time audio processing, the system including:
a data storage device that stores instructions for improved real-time audio processing; and
a processor configured to execute the instructions to perform a method including:
receiving audio data including a plurality of frames having a plurality of frequency bins;
calculating, for each frequency bin, an approximate speech signal estimation based on the plurality of frames;
calculating, for each approximate speech signal estimation, a clean speech estimation and at least one additional target including an ideal ratio mask using a trained neural network model; and
calculating, for each frequency bin, a final clean speech estimation using the calculated at least one additional target including the calculated ideal ratio mask and the calculated clean speech estimation.

### Claim 18 (depends on 17)

The system according to claim 17, wherein the processor is further configured to execute the instructions to perform the method including:
calculating, for each frequency bin, a prior signal-to-noise ratio and a posterior signal-to-noise ratio based on the plurality of frames; and
calculating, for each frequency bin, a suppression gain based on the calculated prior signal-to-noise ratio and the calculated posterior signal-to-noise ratio,
wherein calculating, for each frequency bin, the approximate speech signal estimation based on the plurality of frames includes:
calculating, for each frequency bin, the approximate speech signal estimation based on the calculated suppression gain and the ideal ratio mask.

### Claim 19

A computer-readable storage device storing instructions that, when executed by a computer, cause the computer to perform a method for improved real-time audio processing, the method including:
receiving audio data including a plurality of frames having a plurality of frequency bins;
calculating, for each frequency bin, an approximate speech signal estimation based on the plurality of frames;
calculating, for each approximate speech signal estimation, a clean speech estimation and at least one additional target including an ideal ratio mask using a trained neural network model; and
calculating, for each frequency bin, a final clean speech estimation using the calculated at least one additional target including the calculated ideal ratio mask and the calculated clean speech estimation.

### Claim 20 (depends on 19)

The computer-readable storage device according to claim 19, wherein the method further comprises:
calculating, for each frequency bin, a prior signal-to-noise ratio and a posterior signal-to-noise ratio based on the plurality of frames; and
calculating, for each frequency bin, a suppression gain based on the calculated prior signal-to-noise ratio and the calculated posterior signal-to-noise ratio,
wherein calculating, for each frequency bin, the approximate speech signal estimation based on the plurality of frames includes:
calculating, for each frequency bin, the approximate speech signal estimation based on the calculated suppression gain and the ideal ratio mask.
