# US 9,984,702 B2 · Extraction of reverberant sound using microphone arrays

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,984,702 B2
- Document ID: US9984702B2
- Status: Granted
- Kind: B2
- Application number: 15/178,530
- Filed: 2016-06-09
- Published: 2016-10-06
- Granted: 2018-05-29
- Earliest claimed: 2013-12-11
- Assignees: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Applicants: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Inventors: Oliver Thiergart; Emanuel Habets
- Web page: https://patentgrep.com/patent/US9984702B2

## CPC

- G10L 21/0264: characterised by the type of parameter measurement, e.g. correlation techniques, zero crossing techniques or predictive techniques
- G10L 19/26: Pre-filtering or post-filtering
- G10L 21/02: Speech enhancement, e.g. noise reduction or echo cancellation (reducing echo effects in line transmission systems H04B3/20; echo suppression in hands-free telephones H04M9/08)
- G10L 21/0208: Noise filtering
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04R 5/027: Spatial or constructional arrangements of microphones, e.g. in dummy heads
- G10L 2021/02082: {the noise being echo, reverberation of the speech}
- G10L 2021/02166: {Microphone arrays; Beamforming}
- G11B 2020/10601: {surround sound signal}
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

A method includes estimating a spatial coherence between a first diffuse sound portion in a first microphone signal and a second diffuse sound portion in a second microphone signal. The first microphone signal is captured by a first microphone and the second microphone signal is captured by a second microphone which is spaced apart from the first microphone in a known manner. The method further includes defining a linear constraint for filter coefficients of a diffuse sound filter, the linear constraint being based on the spatial coherence. The method also includes calculating at least one of signal statistics and noise statistics over the first microphone signal and the second microphone signal. The method also includes determining the filter coefficients of the diffuse sound filter by solving an optimization problem concerning at least one of the signal statistics and noise statistics while considering the linear constraint for the filter coefficients.

## Claims

### Claim 1

A method for obtaining filter coefficients for a diffuse sound filter, the method comprising:
defining a linear constraint for the filter coefficients of the diffuse sound filter, the linear constraint being based on a spatial coherence between a first diffuse sound portion in a first microphone signal and a second diffuse sound portion in a second microphone signal, the first microphone signal being captured by a first microphone and the second microphone signal being captured by a second microphone spaced apart from the first microphone;
calculating at least one of a direction of arrival of at least one direct sound, signal statistics over the first and second microphone signals, and noise statistics over the first and second microphone signals;
determining the filter coefficients of the diffuse sound filter by solving an optimization problem concerning at least one of the direction of arrival of the at least one direct sound, the signal statistics, and the noise statistics while considering the linear constraint for the filter coefficients; and
using the filter coefficients within the diffuse sound filter for extracting a diffuse sound from or for suppressing a direct sound in a recording comprising the first microphone signal and the second microphone signal,
wherein one or more of the defining, the calculating, the determining, and the using is implemented, at least in part, by one or more hardware elements of an audio signal processing device.

### Claim 2 (depends on 1)

The method according to claim 1, further comprising
providing the spatial coherence on the basis of a relative transfer function or a correlation of the diffuse sound between the first microphone and the second microphone.

### Claim 3 (depends on 1)

The method according to claim 1,
wherein the spatial coherence is based on a prior measurement of a relative transfer function or a correlation of the diffuse sound for a given environment during time periods in which no direct sound is present in the environment.

### Claim 4 (depends on 1)

The method according to claim 1,
wherein the spatial coherence is based on an a priori information assuming a specific diffuse sound field having assumed properties regarding a correlation of the diffuse sound between the first microphone and the second microphone.

### Claim 5 (depends on 1)

The method according to claim 1, wherein the optimization problem is expressed by
w
m
⁡
(
k
,
n
)
=
argmin
w
⁢
⁢
J
⁡
(
w
)
subject to the linear constraint
wHbm(k,n)=1,
wherein
w(k,n) is a vector of the filter coefficients of the diffuse sound filter;
wm(k,n) is a solution of the optimization problem based on an evaluation of a microphone signal at the m-th microphone;
J(w) is a cost function;
bm(k,n) is a vector of estimated spatial coherences, wherein the m′-th element of the vector is an estimated spatial coherence of the diffuse sound between the m-th microphone and the m′-th microphone;
k is a frequency domain index; and
n is a time domain index.

### Claim 6 (depends on 5)

The method according to claim 5, wherein the cost function J(w) is based on one of noise statistics, a noise power spectral density (PSD) matrix, signal statistics, or a microphone power spectral density (PSD) matrix.

### Claim 7 (depends on 1)

The method according to claim 1, further comprising
estimating at least one of a direction of arrival of at least one direct sound or a relative transfer function of the at least one direct sound between the first microphone and the second microphone;
computing at least one direct sound constraint using the direction of arrival or the relative transfer function of the at least one direct sound, wherein the at least one direct sound constraint results in a suppression of the at least one direct sound.

### Claim 8 (depends on 1)

The method according to claim 1, wherein a solution to the optimization problem is
w
m
⁡
(
k
,
n
)
=
ϕ
d
⁡
(
k
,
n
)
β
+
1
⁢
Φ
x
-
1
⁡
(
k
,
n
)
⁢
b
m
with
β=α(ϕdbmHΦx−1bm)
wherein
wm(k,n) is a solution of the optimization problem based on an evaluation of a microphone signal at the m-th microphone;
bm(k,n) is a vector of estimated spatial coherences, wherein the m′-th element of the vector is an estimated spatial coherence of the diffuse sound between the m-th microphone and the m′-th microphone;
αϵ[0,1] is a user-defined control parameter by which the diffuse sound filter can be scaled between a minimum mean square error spatial filter and a filter that minimizes the output power while satisfying the diffuse sound constraint;
ϕd is a diffuse sound power; and
Φx is a power spectrum matrix of the microphone signals.

### Claim 9 (depends on 8)

The method according to claim 8, further comprising
estimating the diffuse sound power ϕd on the basis of an auxiliary diffuse sound filter.

### Claim 10 (depends on 9)

The method according to claim 9, wherein estimating the diffuse sound power ϕd is performed based on
ϕ
^
d
⁡
(
k
,
n
)
=
w
1
H
⁡
(
k
,
n
)
⁡
[
Φ
x
⁡
(
k
,
n
)
-
Φ
n
⁡
(
k
,
n
)
]
⁢
w
1
⁡
(
k
,
n
)
w
1
H
⁡
(
k
,
n
)
⁢
Γ
d
⁢
w
1
⁡
(
k
,
n
)
wherein
{circumflex over (ϕ)}d is the estimated diffuse sound power which provides an estimate for the diffuse sound power ϕd;
w1 is a vector of filter coefficients of the auxiliary diffuse sound filter;
Φx is a power spectral density matrix of the microphone signals;
Φn is a power spectral density matrix of the noise within the microphone signals; and
Γd is a spatial coherence matrix for the diffuse sound, the (m,m′)-th element of Γd being a spatial coherence γm′,m between microphone m and m′.

### Claim 11 (depends on 1)

The method according to claim 1, further comprising:
wherein the using the filter coefficients comprises performing a linear combination of the first microphone signal and the second microphone signal using the filter coefficients of the diffuse sound filter.

### Claim 12

A non-transitory digital storage medium having stored thereon a computer program for performing a method for obtaining filter coefficients for a diffuse sound filter, the method comprising:
defining a linear constraint for the filter coefficients of the diffuse sound filter, the linear constraint being based on a spatial coherence between a first diffuse sound portion in a first microphone signal and a second diffuse sound portion in a second microphone signal, the first microphone signal being captured by a first microphone and the second microphone signal being captured by a second microphone spaced apart from the first microphone;
calculating at least one of a direction of arrival of at least one direct sound, signal statistics over the first and second microphone signals, and noise statistics over the first and second microphone signals;
determining the filter coefficients of the diffuse sound filter by solving an optimization problem concerning at least one of the direction of arrival of the at least one direct sound, the signal statistics, and the noise statistics while considering the linear constraint for the filter coefficients; and
using the filter coefficients within the diffuse sound filter for extracting a diffuse sound from or for suppressing a direct sound in a recording comprising the first microphone signal and the second microphone signal,
when said computer program is run by a computer.

### Claim 13

An apparatus for obtaining filter coefficients for a diffuse sound filter, the apparatus comprising:
a linear constraint calculator configured to define a linear constraint for the filter coefficients of the diffuse sound filter, the linear constraint being based on a spatial coherence between a first diffuse sound portion in a first microphone signal and a second diffuse sound portion in a second microphone signal, the first microphone signal being captured by a first microphone and the second microphone signal being captured by a second microphone spaced apart from the first microphone;
a calculator configured to calculate at least one of a direction of arrival of at least one direct sound, signal statistics over the first and second microphone signals, and noise statistics over the first and second microphone signals; and
a filter coefficients calculator configured to determine the filter coefficients of the diffuse sound filter by solving an optimization problem concerning at least one of the direction of arrival of the at least one direct sound, the signal statistics, and the noise statistics while considering the linear constraint for the filter coefficients,
wherein the apparatus is configured for using the filter coefficients within the diffuse sound filter for extracting a diffuse sound from or for suppressing a direct sound in a recording comprising the first microphone signal and the second microphone signal, and
wherein one or more of the linear constraint calculator, the calculator, and the filter coefficients calculator is implemented, at least in part, by one or more hardware elements of the apparatus.

### Claim 14 (depends on 13)

The apparatus according to claim 13, wherein the spatial coherence is based on a relative transfer function of the diffuse sound between the first microphone and the second microphone.

### Claim 15 (depends on 13)

The apparatus according to claim 13, further comprising
an auxiliary filter coefficients calculator configured to determine auxiliary filter coefficients of an auxiliary diffuse sound filter by solving a different optimization problem while considering a different linear constraint for the auxiliary filter coefficients than the filter coefficients calculator;
the auxiliary diffuse sound filter being configured to estimate an estimated diffuse sound power;
wherein the filter coefficients calculator is configured to take into account the estimated diffuse sound power when determining the filter coefficients of the diffuse sound filter.

### Claim 16 (depends on 1)

The method according to claim 1, wherein the using the filter coefficients within the diffuse sound filter comprises multiplying the first microphone signal and the second microphone signal by the filter coefficients to obtain an estimate of a diffuse sound in the recording.