# US 9,439,019 B2 · Sound signal processing method and apparatus

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 9,439,019 B2
- Document ID: US9439019B2
- Status: Granted
- Kind: B2
- Application number: 14/822,476
- Filed: 2015-08-10
- Published: 2016-03-03
- Granted: 2016-09-06
- Earliest claimed: 2014-08-29
- Assignees: Huawei Technologies Co., Ltd.
- Applicants: Huawei Technologies Co., Ltd.
- Inventors: Jingdong Chen; Haiting Li; Deming Zhang
- Web page: https://patentgrep.com/patent/US9439019B2

## CPC

- H04S 3/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- H04R 3/005: {for combining the signals of two or more microphones (specially adapted for hearing aids H04R25/407)}
- H04R 5/027: Spatial or constructional arrangements of microphones, e.g. in dummy heads
- H04S 3/02: of the matrix type, i.e. in which input signals are combined algebraically, e.g. after having been phase shifted with respect to each other
- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- H04W 88/02: Terminal devices
- H04R 1/406: {microphones}
- H04R 2201/401: 2D or 3D arrays of transducers
- H04R 2201/405: Non-uniform arrays of transducers or a plurality of uniform arrays with different transducer spacing
- H04R 2203/12: Beamforming aspects for stereophonic sound reproduction with loudspeaker arrays
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/13: Application of wave-field synthesis in stereophonic audio systems
- H04S: STEREOPHONIC SYSTEMS
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04W: WIRELESS COMMUNICATION NETWORKS (broadcast communication H04H; communication systems using wireless links for non-selective communication, e.g. wireless extensions H04M1/72)

## Abstract

A sound signal processing method and apparatus are provided that relate to the audio signal processing field. The method in the present invention includes acquiring, by a mobile terminal, sound signals from a three-dimensional sound field, where at least three microphones are disposed on the mobile terminal and one microphone is configured to receive a sound signal in at least one direction; acquiring, according to the acquired sound signals, a direction of a sound source relative to the mobile terminal; and obtaining spatial audio signals according to the direction of the sound source relative to the mobile terminal and the acquired sound signals, where the spatial audio signals are used for simulating the three-dimensional sound field. The present invention is applicable to a process of collecting and processing signals in a three-dimensional sound field surrounding a terminal.

## Claims

### Claim 1

A sound signal processing method, comprising:
acquiring, by a mobile terminal, sound signals from a three-dimensional sound field, wherein at least three microphones are disposed on the mobile terminal and one microphone is configured to receive a sound signal in at least one direction;
acquiring, according to the acquired sound signals, a direction of a sound source relative to the mobile terminal; and
obtaining spatial audio signals according to the direction of the sound source relative to the mobile terminal and the acquired sound signals, wherein the spatial audio signals are used for simulating the three-dimensional sound field,
wherein acquiring, according to the acquired sound signals, the direction of the sound source relative to the mobile terminal comprises:
acquiring a time difference of arrival between a sound signal in one direction and a sound signal in another direction that are received by the microphones on the mobile terminal, wherein the microphones on the mobile terminal are configured to receive the sound signals in at least four directions; and
obtaining, according to the acquired time difference of arrival and locations of the microphones on the mobile terminal, the direction of the sound source relative to the mobile terminal,
wherein acquiring the time difference of arrival between the sound signal in one direction and the sound signal in another direction that are received by the microphones on the mobile terminal comprises acquiring, according to
τ
^
m
=
argmax
p
⁢
r
y
1
⁢
y
m
⁡
(
p
)
,
a time difference {circumflex over (τ)}m of arrival between a sound signal received by an mth microphone and a sound signal received by a 1st microphone,
wherein p represents a quantity of delay sampling points, y1(n) represents the sound signal received by the 1st microphone, ym(n) represents the sound signal received by the mth microphone,
r
y
1
⁢
y
m
⁡
(
p
)
=
∫
-
∞
∞
⁢
Φ
⁡
(
f
)
⁢
S
y
1
⁢
y
m
⁡
(
f
)
⁢
ⅇ
j2π
⁢
⁢
f
⁢
⁢
ρ
⁢
⁢
ⅆ
f
,
Sy1ym(f) represents a correlated spectrum between y1(n) and ym(n), and Φ(f) represents a weight function of a frequency domain in which the sound signals received in all directions are located, and
wherein obtaining, according to the acquired time difference of arrival and locations of the microphones on the mobile terminal, the direction of the sound source relative to the mobile terminal comprises:
estimating, according to
θ
^
2
=
arcsin
⁡
(
τ
^
2
⁢
c
h
)
,
a direction {circumflex over (θ)}2 of a sound signal received by a 2nd microphone;
estimating, according to
θ
^
3
=
arccos
⁡
(
τ
^
3
⁢
c
d
2
+
h
2
)
-
α
,
a direction {circumflex over (θ)}3 of a sound signal received by a 3rd microphone;
estimating, according to
θ
^
4
=
arccos
⁡
(
τ
^
4
⁢
c
d
)
,
a direction {circumflex over (θ)}4 of a sound signal received by a 4th microphone; and
acquiring, according to
θ
=
1
3
⁢
∑
m
=
2
4
⁢
⁢
θ
^
m
,
the direction θ of the sound source relative to the mobile terminal,
wherein the 1st microphone and the 4th microphone are located on a same side and the 1st microphone and the 2nd microphone are located on a same side, and
wherein c represents a sound speed, d represents a distance between a central point of the 1st microphone and a central point of the 4th microphone, d is equal to a distance between a central point of the 2nd microphone and a central point of the 3rd microphone, h represents a distance between the central point of the 1st microphone and the central point of the 2nd microphone, h is equal to a distance between the central point of the 3rd microphone and the central point of the 4th microphone, and α represents an included angle between a diagonal of the mobile terminal and a side of an angle at which the 1st microphone and the 4th microphone are located.

### Claim 2

A sound signal processing method, comprising:
acquiring, by a mobile terminal, sound signals from a three-dimensional sound field, wherein at least three microphones are disposed on the mobile terminal and one microphone is configured to receive a sound signal in at least one direction;
acquiring, according to the acquired sound signals, a direction of a sound source relative to the mobile terminal; and
obtaining spatial audio signals according to the direction of the sound source relative to the mobile terminal and the acquired sound signals, wherein the spatial audio signals are used for simulating the three-dimensional sound field,
wherein acquiring, according to the acquired sound signals, the direction of the sound source relative to the mobile terminal comprises:
acquiring a time difference of arrival between a sound signal in one direction and a sound signal in another direction that are received by the microphones on the mobile terminal, wherein the microphones on the mobile terminal are configured to receive the sound signals in at least four directions; and
obtaining, according to the acquired time difference of arrival and locations of the microphones on the mobile terminal, the direction of the sound source relative to the mobile terminal,
wherein acquiring the time difference of arrival between the sound signal in one direction and the sound signal in another direction that are received by the microphones on the mobile terminal comprises acquiring, according to
τ
^
m
=
argmax
p
⁢
r
y
1
⁢
y
m
⁡
(
p
)
,
a time difference {circumflex over (τ)}m of arrival between a sound signal received by an mth microphone and a sound signal received by a 1st microphone,
wherein p represents a quantity of delay sampling points, y1(n) represents the sound signal received by the 1st microphone, ym(n) represents the sound signal received by the mth microphone,
r
y
1
⁢
y
m
⁡
(
p
)
=
∫
-
∞
∞
⁢
Φ
⁡
(
f
)
⁢
S
y
1
⁢
y
m
⁡
(
f
)
⁢
ⅇ
j2π
⁢
⁢
f
⁢
⁢
ρ
⁢
⁢
ⅆ
f
,
Sy1ym(f) represents a correlated spectrum between y1(n) and ym(n), and Φ(f) represents a weight function of a frequency domain in which the sound signals received in all directions are located, and
wherein obtaining, according to the acquired time difference of arrival and locations of the microphones on the mobile terminal, the direction of the sound source relative to the mobile terminal comprises:
acquiring, according to
{
τ
^
2
=
(
y
2
⁢
cos
⁢
⁢
θ
-
x
2
⁢
sin
⁢
⁢
θ
)
-
(
y
1
⁢
cos
⁢
⁢
θ
-
x
1
⁢
sin
⁢
⁢
θ
)
c
τ
^
3
=
(
y
3
⁢
cos
⁢
⁢
θ
-
x
3
⁢
sin
⁢
⁢
θ
)
-
(
y
1
⁢
cos
⁢
⁢
θ
-
x
1
⁢
sin
⁢
⁢
θ
)
c
τ
^
4
=
(
y
4
⁢
cos
⁢
⁢
θ
-
x
4
⁢
sin
⁢
⁢
θ
)
-
(
y
1
⁢
cos
⁢
⁢
θ
-
x
1
⁢
sin
⁢
⁢
θ
)
c
,
the direction θ of the sound source relative to the mobile terminal,
wherein the 1st microphone and the 2nd microphone are located on an axis of symmetry of one surface of the mobile terminal and are separately located on two parallel edges on the one surface,
wherein the 3rd microphone and the 4th microphone are located on an axis of symmetry of another surface of the mobile terminal and are separately located on two parallel edges on the another surface, and
wherein c represents a sound speed, (x1,y1) represents coordinates corresponding to the 1st microphone, (x2,y2) represents coordinates corresponding to the 2nd microphone, (x3,y3) represents coordinates corresponding to the 3rd microphone, and (x4,y4) represents coordinates corresponding to the 4th microphone.

### Claim 3

A sound signal processing method, comprising:
acquiring, by a mobile terminal, sound signals from a three-dimensional sound field, wherein at least three microphones are disposed on the mobile terminal and one microphone is configured to receive a sound signal in at least one direction:
acquiring, according to the acquired sound signals, a direction of a sound source relative to the mobile terminal; and
obtaining spatial audio signals according to the direction of the sound source relative to the mobile terminal and the acquired sound signals,
wherein the spatial audio signals are used for simulating the three-dimensional sound field,
wherein the spatial audio signals comprise at least a signal in a left direction, a signal in a right direction, a signal in a center direction, a left rear surround signal, and a right rear surround signal, and
wherein obtaining spatial audio signals according to the direction of the sound source relative to the mobile terminal and the acquired sound signals comprises:
generating gain adjusting parameters by using the direction of the sound source relative to the mobile terminal, wherein the gain adjusting parameters comprise a first-type gain adjusting parameter αF corresponding to the signal in the left direction, the signal in the right direction, and the signal in the center direction, a first-type gain adjusting parameter αB corresponding to the left rear surround signal and the right rear surround signal, a second-type gain adjusting parameter αL corresponding to the signal in the left direction and the left rear surround signal, and a second-type gain adjusting parameter αR corresponding to the signal in the right direction and the right rear surround signal, and in addition, for the direction of the sound source relative to the mobile terminal, when θε[0,90)∪(270,360], αF>αB, when θε(90,270), αF<αB, when θε(0,180), αL>αR, and when θε(180,360), αL<αR;
acquiring initial values of the signals in all the directions in the spatial audio signals according to the sound signals; and
generating the signal in the left direction, the signal in the right direction, the signal in the center direction, the left rear surround signal, and the right rear surround signal according to the gain adjusting parameters and the initial values of the signals in all the directions in the spatial audio signals.

### Claim 4 (depends on 3)

The sound signal processing method according to claim 3, wherein the spatial audio signals further comprise a two-channel signal, and wherein the method further comprises downmixing the signal in the left direction, the signal in the right direction, the signal in the center direction, the left rear surround signal, and the right rear surround signal to generate the two-channel signal.

### Claim 5 (depends on 3)

The sound signal processing method according to claim 3, wherein generating the gain adjusting parameters by using the direction of the sound source relative to the mobile terminal comprises:
generating, according to
α
F
=
{
α
F
,
max
⁢
⁢
1
,
θ
=
0
α
F
,
max
⁢
⁢
1
+
α
F
,
med
⁢
⁢
1
-
α
F
,
max
⁢
⁢
1
90
⁢
θ
,
θ
∈
(
0
,
90
)
α
F
,
med
⁢
⁢
1
,
θ
=
90
α
F
,
med
⁢
⁢
1
+
α
F
,
min
-
α
F
,
med
⁢
⁢
1
90
⁢
(
θ
-
90
)
,
θ
∈
(
90
,
180
)
α
F
,
min
,
θ
=
180
α
F
,
min
+
α
F
,
med
⁢
⁢
2
-
α
F
,
min
90
⁢
(
θ
-
180
)
,
θ
∈
(
180
,
270
)
α
F
,
med
⁢
⁢
2
,
θ
=
270
α
F
,
med
⁢
⁢
2
+
α
F
,
max
⁢
⁢
2
-
α
F
,
med
⁢
⁢
2
90
⁢
(
θ
-
270
)
,
θ
∈
(
270
,
360
)
,
the first-type gain adjusting parameter αF corresponding to the signal in the left direction, the signal in the right direction, and the signal in the center direction;
generating, according to
α
B
=
{
α
B
,
min
⁢
⁢
1
,
θ
=
0
α
B
,
min
⁢
⁢
1
+
α
B
,
med
⁢
⁢
1
-
α
B
,
min
⁢
⁢
1
90
⁢
θ
,
θ
∈
(
0
,
90
)
α
B
,
med
⁢
⁢
1
,
θ
=
90
α
B
,
med
⁢
⁢
1
+
α
B
,
max
-
α
B
,
med
⁢
⁢
1
90
⁢
(
θ
-
90
)
,
θ
∈
(
90
,
180
)
α
B
,
max
,
θ
=
180
α
B
,
max
+
α
B
,
med
⁢
⁢
2
-
α
B
,
max
90
⁢
(
θ
-
180
)
,
θ
∈
(
180
,
270
)
α
B
,
med
⁢
⁢
2
,
θ
=
270
α
B
,
med
⁢
⁢
2
+
α
B
,
min
⁢
⁢
2
-
α
B
,
med
⁢
⁢
2
90
⁢
(
θ
-
270
)
,
θ
∈
(
270
,
360
)
,
the first-type gain adjusting parameter αB corresponding to the left rear surround signal and the right rear surround signal, wherein αF, min, αF, med1, αF, max1, αF, med2, αF, max2, αB, min1, αB, med1, αB, min2, αB, med2, and αB, max are constants greater than zero, and αF, min<αF, med1<αF, max1, αF, min<αF, med2<αF, max2, αB, min1<αB, med1<αB, max, αB, min2<αB, med2<αB, max, αB, med1=αF, med1, and αB, med2=αF, med2;
generating, according to
α
L
=
{
α
L
,
med
⁢
⁢
1
,
θ
=
0
α
L
,
med
⁢
⁢
1
+
α
L
,
max
-
α
L
,
med
⁢
⁢
1
90
⁢
θ
,
θ
∈
(
0
,
90
)
α
L
,
max
,
θ
=
90
α
L
,
max
+
α
L
,
med
⁢
⁢
2
-
α
L
,
max
90
⁢
(
θ
-
90
)
,
θ
∈
(
90
,
180
)
α
L
,
med
⁢
⁢
2
,
θ
=
180
α
L
,
med
⁢
⁢
2
+
α
L
,
min
-
α
L
,
med
⁢
⁢
2
90
⁢
(
θ
-
180
)
,
θ
∈
(
180
,
270
)
α
L
,
min
,
θ
=
270
α
L
,
min
+
α
L
,
med
⁢
⁢
1
-
α
L
,
min
90
⁢
(
θ
-
270
)
,
θ
∈
(
270
,
360
)
,
the second-type gain adjusting parameter αL corresponding to the signal in the left direction and the left rear surround signal; and
generating, according to
α
R
=
{
α
R
,
med
⁢
⁢
1
,
θ
=
0
α
R
,
med
⁢
⁢
1
+
α
R
,
min
-
α
R
,
med
⁢
⁢
1
90
⁢
θ
,
θ
∈
(
0
,
90
)
α
R
,
min
,
θ
=
90
α
R
,
min
+
α
R
,
med
⁢
⁢
2
-
α
R
,
min
90
⁢
(
θ
-
90
)
,
θ
∈
(
90
,
180
)
α
R
,
med
⁢
⁢
2
,
θ
=
180
α
R
,
med
⁢
⁢
2
+
α
R
,
max
-
α
R
,
med
⁢
⁢
2
90
⁢
(
θ
-
180
)
,
θ
∈
(
180
,
270
)
α
R
,
max
,
θ
=
270
α
R
,
max
+
α
R
,
med
⁢
⁢
1
-
α
R
,
max
90
⁢
(
θ
-
270
)
,
θ
∈
(
270
,
360
)
,
the second-type gain adjusting parameter αR corresponding to the signal in the right direction and the right rear surround signal, wherein αL, med1, αL, max, αL, med2, αL, min, αR, med1, αR, min, αR, med2, and αR, max are constants greater than zero, αL, min<αL, med1<αL, max, αL, min<αL, med2<αL, max, αR, min<αR, med1<αR, max, αR, min<αR, med2<αR, max, αL, med1=αR, med1, and αL, med2=αR, med2.

### Claim 6 (depends on 5)

The sound signal processing method according to claim 5, wherein generating the signals in all the directions in the spatial audio signals according to the gain adjusting parameters and the initial values of the signals in all the directions in the spatial audio signals comprises:
acquiring the signal SL, final(n) in the left direction, the signal SC, final(n) in the center direction, the signal SR, final(n) in the right direction, the left rear surround direction signal SLS, final(n), and the right rear surround signal SRS, final(n) according to
{
S
L
,
final
⁡
(
n
)
=
S
L
⁡
(
n
)
*
α
F
*
α
L
S
C
,
final
⁡
(
n
)
=
S
C
⁡
(
n
)
*
α
F
S
R
,
final
⁡
(
n
)
=
S
R
⁡
(
n
)
*
α
F
*
α
R
S
LS
,
final
⁡
(
n
)
=
S
LS
⁡
(
n
)
*
α
B
*
α
L
S
RS
,
final
⁡
(
n
)
=
S
RS
⁡
(
n
)
*
α
B
*
α
R
,
wherein SL(n) represents the initial value of the signal in the left direction, SC(n) represents an initial value of the signal in the left direction, SR(n) represents an initial value of the signal in the right direction, SLS(n) represents an initial value of the left rear surround signal, SRS(n) represents an initial value of the right rear surround signal, and n represents a sampling point number.

### Claim 7 (depends on 5)

The sound signal processing method according to claim 5, wherein
acquiring the initial values of the signals in all the directions in the spatial audio signals according to the sound signals comprises acquiring beams BFL(n), BFR(n), BFLS(n), and BFRS(n) with heart-shaped directivity, wherein a maximum directivity of BFL(n) is within (0,90°) a maximum directivity of BFLS(n) is within (90°,180°), a maximum directivity of BFRS(n) is within (180°,270°), and a maximum directivity of BFR(n) is within (270°,360°), wherein an initial value of the signal in the left direction SL(n)=BFL(n), an initial value of the signal in the center direction
S
C
⁡
(
n
)
=
BF
L
⁡
(
n
)
+
BF
R
⁡
(
n
)
2
,
an initial value of the signal in the right direction SR(n)=BFR(k), an initial value of the left rear surround signal SLS(n)=BFLS(n), and an initial value of the right rear surround signal SRS(n)=BFRS(n), wherein SL(n) represents the initial value of the signal in the left direction, SC(n) represents the initial value of the signal in the center direction, SR(n) represents the initial value of the signal in the right direction, SLS(n) represents the initial value of the left rear surround signal, SRS(n) represents the initial value of the right rear surround signal, and n represents the sampling point number.

### Claim 8 (depends on 7)

The sound signal processing method according to claim 7, wherein acquiring beams BFL(n), BFR(n), BFLS(n), and BFRS(n) with the heart-shaped directivity comprises:
acquiring a beam BFij(n) with heart-shaped directivity by using the sound signals received by the 1st to the 4th microphones, wherein a point of a maximum directivity of BFij(n) is in a direction pointing from a connecting line between the ith microphone and a jth microphone to the ith microphone, and a zero point of directivity of BFij(n) is in a direction pointing from the connecting line between the ith microphone and the jth microphone to the jth microphone, wherein i and j are equal to 1, 2, 3, or 4; and
acquiring the beams BFL(n), BFR(n), BFLS(n), and BFRS(n) with heart-shaped directivity according to
{
BF
L
⁡
(
n
)
=
BF
21
⁡
(
n
)
+
BF
23
⁡
(
n
)
2
BF
R
⁡
(
n
)
=
BF
12
⁡
(
n
)
+
BF
14
⁡
(
n
)
2
BF
LS
⁡
(
n
)
=
BF
32
⁡
(
n
)
+
BF
34
⁡
(
n
)
2
BF
RS
⁡
(
n
)
=
BF
41
⁡
(
n
)
+
BF
43
⁡
(
n
)
2
.

### Claim 9 (depends on 5)

The sound signal processing method according to claim 5, wherein generating the signals in all the directions in the spatial audio signals according to the gain adjusting parameters and the initial values of the signals in all the directions in the spatial audio signals comprises:
acquiring the signal SL, final(n) in the left direction according to SL, final(n)=SL(n)*αL;
acquiring the signal SC, final(n) in the center direction according to SC, final(n)=SC(n);
acquiring the signal SR, final(n) in the right direction according to SR, final(n)=SR(n)*αR;
acquiring the left rear surround signal SLS, final(n) according to SLS, final(n)=SLS(n)*αL; and
acquiring the right rear surround signal SRS, final(n) according to SRS, final(n)=SRS(n)*αR,
wherein SL(n) represents an initial value of the signal in the left direction, SC(n) represents an initial value of the signal in the center direction, SR(n) represents an initial value of the signal in the right direction, SLS(n) represents an initial value of the left rear surround signal, SRS(n) represents an initial value of the right rear surround signal, and n represents the sampling point number.