# US 11,308,973 B2 · Method for processing multi-channel audio signal on basis of neural network and electronic device

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,308,973 B2
- Document ID: US11308973B2
- Status: Granted
- Kind: B2
- Application number: 16/987,475
- Filed: 2020-08-07
- Published: 2021-02-11
- Granted: 2022-04-19
- Earliest claimed: 2019-08-07
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Soonho Baek; Kiho Cho; Jongmo Keum; Beakkwon Son; Myoungho Lee; Yonghoon Lee; Hangil Moon; Jaemo Yang; Gunwoo Lee
- Web page: https://patentgrep.com/patent/US11308973B2

## CPC

- G10L 21/0232: Processing in the frequency domain
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 21/0208: Noise filtering
- G10L 21/0364: for improving intelligibility
- G10L 25/30: using neural networks
- G10L 25/51: for comparison or discrimination
- H04R 1/406: {microphones}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04R 29/005: {Microphone arrays}
- H04S 3/008: {in which the audio signals are in digital form, i.e. employing more than two discrete digital channels (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- G10L 2021/02166: {Microphone arrays; Beamforming}
- H04R 1/1083: {Reduction of ambient noise (active noise reduction per se G10K11/175; protective devices for the ear, e.g. providing acoustic protection A61F11/06)}
- H04R 2410/01: Noise reduction using microphones having different directional characteristics
- H04R 2430/03: Synergistic effects of band splitting and sub-band processing
- H04R 2499/11: Transducers incorporated or for use in hand-held devices, e.g. mobile phones, PDA's, camera's
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

According to an embodiment, an electronic device may include a memory configured to store a noise removal neural network model and data utilized in the noise removal neural network model and a processor electrically connected to the memory wherein the memory may store instructions that, when executed, enable the processor to: output a first channel signal using a first beamformer for a multi-channel audio signal; output a second channel signal using a second beamformer; generate a third channel signal that compensates for a difference in noise levels between the first channel signal and the second channel signal; and train the noise removal neural network model by using the third channel signal in which the difference in noise levels is compensated for and the first channel signal as input values.

## Claims

### Claim 1

An electronic device comprising:
a communication circuit;
at least two microphones;
a processor configured to be operatively connected to the communication circuit and the at least two microphones; and
a memory configured to be operatively connected to the processor and store a noise removal neural network model using a first channel signal corresponding to a first beamformer and a second channel signal corresponding to a second beamformer as an input value,
wherein the memory stores instructions that, when executed, enable the processor to:
acquire the first channel signal using the first beamformer and the second channel signal using the second beamformer for an audio input signal input through the at least two microphones;
generate a third channel signal in which a difference in noise levels between the first channel signal and the second channel signal is compensated; and
output a neural network output signal by using the first channel signal and the third channel signal as input values for the noise removal neural network model.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the memory further stores instructions that, when executed, enable the processor to:
identify sound sections and silent sections from the first channel signal and the second channel signal; and
compare a difference in output levels between the silent section of the first channel signal and the silent section of the second channel signal to calculate the difference in noise levels.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the memory further stores instructions that, when executed, enable the processor to:
estimate an average value of the difference in noise levels; and
frequency-compensate the second channel signal with a compensation value corresponding to the average value to generate the third channel signal.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the first channel signal includes a signal in which a speaker's voice level is emphasized and noise is attenuated using a difference in arrival times of multi-channel voices and a noise removal rate, and the second channel signal includes a signal in which the speaker's voice level is attenuated.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the neural network output signal includes a signal with an improved signal-to-noise-ratio (SNR) of a voice compared to a noise signal or a voice signal from which noise is removed via the noise removal neural network model,
wherein the memory further stores instructions that, when executed, enable the processor to:
identify a sound section of the first channel signal,
compare the first channel signal in the sound section and the neural network output signal to estimate an average value of a difference in signal level for the sound section; and
process the neural network output signal in order to compensate for voice distortion based on the average value of the difference in signal level in the sound section.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the memory further stores instructions that, when executed, enable the processor to:
identify a sound section and a silent section of the second channel signal,
estimate a noise removal amount from frequency components of the silent section of the second channel signal; and
estimate a noise removal amount of the sound section of the second channel signal.

### Claim 7 (depends on 6)

The electronic device of claim 6, wherein the memory further stores instructions that, when executed, enable the processor to:
compensate the noise removal amount in the silent section of the second channel signal to be reduced by the noise removal amount in the sound section of the second channel signal; and
process the neural network output signal according to the compensated noise removal amount in the silent section of the second channel signal so that a noise removal amount of the neural network output signal is uniform.

### Claim 8 (depends on 7)

The electronic device of claim 7, wherein the memory further stores instructions that, when executed, enable the processor to remove or attenuate residual noise in the processed neural network output signal such that the compensated voice distortion and noise removal amount are uniform.

### Claim 9 (depends on 1)

The electronic device of claim 1, wherein the memory further stores instructions that, when executed, enable the processor to:
transmit the audio input signal input through the at least two microphones to the first beamformer and the second beamformer by performing short time Fourier transform (STFT); and
perform inverse short time Fourier transform (ISTFT) on the neural network output signal whose residual noise is removed or attenuated to restore to a sound source signal.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein the noise removal neural network model is a neural network model based on a convolutional neural network (CNN), a recurrent neural network (RNN), and/or a deep neural network (DNN).

### Claim 11

An electronic device comprising:
a memory configured to store a noise removal neural network model and data utilized in the noise removal neural network model; and
a processor configured to be electrically connected to the memory,
wherein the memory stores instructions that, when executed, enable the processor to:
output a first channel signal using a first beamformer for a multi-channel voice signal;
output a second channel signal using a second beamformer;
generate a third channel signal in which a difference in noise levels between the first channel signal and the second channel signal is compensated; and
train the noise removal neural network model by using the third channel signal in which the difference in noise levels is compensated and the first channel signal as input values.

### Claim 12 (depends on 11)

The electronic device of claim 11, wherein the memory stores instructions that, when executed, enable the processor to:
receive the multi-channel voice signal to build the noise removal neural network model that outputs a noise-removed voice signal or a signal with improved signal-to-noise-ratio (SNR) compared to a noise signal; and
train the noise removal neural network model to update a noise removal rule, a noise removal function, and/or a noise removal gain coefficient of the noise removal neural network model.

### Claim 13 (depends on 12)

The electronic device of claim 12, further comprising a communication circuit,
wherein the memory stores instructions that, when executed, enable the processor to transmit the noise removal neural network model, the updated noise removal neural network model, and update information of the noise removal neural network model to an external electronic device through the communication circuit.

### Claim 14 (depends on 11)

The electronic device of claim 11, wherein the first channel signal includes a signal in which a speaker's voice level is emphasized and noise is attenuated using a difference in arrival times of the multi-channel voice signal and a noise removal rate, and
the second channel signal includes a signal in which the speaker's voice level is attenuated.

### Claim 15 (depends on 11)

The electronic device of claim 11, wherein the memory stores instructions that, when executed, enable the processor to:
identify sound sections and silent sections from the first channel signal and the second channel signal; and
compare a difference in output levels between the silent section of the first channel signal and the silent section of the second channel signal to calculate the difference in noise levels.

### Claim 16 (depends on 15)

The electronic device of claim 15, wherein the memory stores instructions that, when executed, enable the processor to:
estimate an average value of an output level difference between the silent section of the first channel signal and the silent section of the second channel signal; and
frequency-compensate a noise signal of the second channel signal with a compensation value corresponding to the average value to generate the third channel signal.

### Claim 17 (depends on 12)

The electronic device of claim 12, wherein the memory stores instructions that, when executed, enable the processor to:
extract a feature vector based on the third channel signal and the first channel signal; and
input the feature vector as an input value for training the noise removal neural network model,
wherein the feature vector includes a first feature value input to a voice main channel, a second feature value input to a noise reference channel, and a third feature value in which a signal-to-noise ratio (SNR) is improved from the first feature value.

### Claim 18 (depends on 17)

The electronic device of claim 17, wherein the memory stores instructions that, when executed, enable the processor to:
designate the first feature value and the second feature value as input variables;
designate the third feature value as an output variable; and
train the noise removal neural network model to derive a noise removal rule, a noise removal function, and/or noise removal gain coefficients based on training data.

### Claim 19

An electronic device comprising:
a communication circuit;
a memory configured to store a noise removal neural network model and data utilized in the noise removal neural network model; and
a processor configured to be electrically connected to the memory,
wherein the memory stores instructions that, when executed, enable the processor to:
receive a first channel user voice signal and a second channel user voice signal from an external electronic device through the communication circuit;
generate a third channel user voice signal in which a difference in noise levels between the first channel user voice signal and the second channel user voice signal is compensated; and
train the noise removal neural network model using the third channel user voice signal in which the difference in noise levels is compensated and the first channel user voice signal as input values to optimize the noise removal neural network model for user voice.

### Claim 20 (depends on 19)

The electronic device of claim 19, wherein the first channel user voice signal includes a signal in which a user's voice level is emphasized and noise is attenuated, and the second channel user voice signal includes a signal in which the user's voice is attenuated, and
wherein the memory stores instructions that, when executed, enable the processor to transmit the noise removal neural network model optimized for user voice and/or update information of the noise removal neural network model to an external electronic device through the communication circuit.