US 12,750,625 B2 · Apparatus, method or computer program for synthesizing a spatially extended sound source using variance or covariance data

Abstract

An apparatus synthesizing a spatially extended sound source, includes: a storage for storing one or more rendering data items for different limited spatial sectors, wherein the different limited spatial sectors are located in a rendering range for a listener, wherein the one or more rendering data items for a limited spatial sector includes at least one of a left variance data item, a right variance data item, and a left-right covariance data item; a sector identification processor for identifying one or more limited spatial sectors for the spatially extended sound source within the rendering range for the listener based on spatially extended sound source data; a target data calculator for calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data; and an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data.

Claims21

Dependent claims

An apparatus for synthesizing a spatially extended sound source (SESS), comprising:

a storage for storing one or more rendering data items for different limited spatial sectors, wherein the different limited spatial sectors are located in a rendering range for a listener, wherein the one or more rendering data items for a limited spatial sector comprises at least one of a left variance data item, a right variance data item, and a left-right covariance data item related to the left head related function data and the right head related function data;
a sector identification processor for identifying one or more limited spatial sectors for the spatially extended sound source within the rendering range for the listener based on spatially extended sound source data;
a target data calculator for calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data;
and
an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data,
wherein the storage is configured to store, for each limited spatial sector, a frequency dependent representation of the left variance data item, a frequency dependent representation of the right variance data item, and a frequency dependent representation of the covariance data item.

3 more elements

The apparatus for synthesizing a spatially extended sound source (SESS), comprising:

a storage for storing one or more rendering data items for different limited spatial sectors, wherein the different limited spatial sectors are located in a rendering range for a listener, wherein the one or more rendering data items for a limited spatial sector comprises at least one of a left variance data item, a right variance data item, and a left-right covariance data item related to the left head related function data and the right head related function data;
a sector identification processor for identifying one or more limited spatial sectors for the spatially extended sound source within the rendering range for the listener based on spatially extended sound source data;
a target data calculator for calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data;
and
an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data,
wherein the target data calculator is configured for calculating, as the target rendering data, at least one of an inter-aural or inter-channel coherence cue, an inter-aural or inter-channel level difference cue, an inter-aural or inter-channel phase difference cue, a first side gain, and a second side gain, and
wherein the audio processor is configured to perform at least one of an inter-channel or inter-aural coherence adjustment, an inter-aural or inter-channel phase difference adjustment, or an inter-aural or inter-channel level difference adjustment using corresponding cues as the target rendering data,
wherein the target data calculator is configured to calculate the inter-aural or inter-channel coherence cue, so that a value of the inter-aural or inter-channel coherence cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel coherence cue
IACC
⁡
(
ω
)
=
Σ
n
=
1
N
⁢
I
⁢
A
⁢
C
⁢
C
g
⁢
r
⁢
id
,
n
(
ω
)
N
w
⁢
e
⁢
i
⁢
g
⁢
h
⁢
t
⁢
e
⁢
d
,
wherein the target data calculator is configured to calculate the inter-aural or inter-channel level difference cue so that a value of the inter-aural or inter-channel level difference cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel level difference cue, or
wherein the target data calculator is configured to calculate the inter-aural or inter-channel phase difference cue so that a value of the inter-aural or inter-channel phase difference cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel phase difference cue,
IALD
⁡
(
ω
)
=
10
⁢
log
1
⁢
0
⁢
E
⁢
{
❘
"\[LeftBracketingBar]"
Y
l
(
ω
)
❘
"\[RightBracketingBar]"
2
}
E
⁢
{
❘
"\[LeftBracketingBar]"
Y
r
(
ω
)
❘
"\[RightBracketingBar]"
2
}
=
10
⁢
log
1
⁢
0
⁢
∑
n
=
1
N
⁢
A
l
,
n
2
∑
n
=
1
N
⁢
A
r
,
n
2
,
(
24
)
IAPD
=
∠
⁡
(
E
⁢
{
Y
l
(
ω
)
·
Y
r
*
(
ω
)
}
)
=
∠
⁡
(
IAC
⁢
C
⁡
(
ω
)
)
=
∠
⁡
(
∑
n
=
1
N
A
l
,
n
·
A
r
,
n
·
e
j
⁡
(
ϕ
l
,
n
-
ϕ
r
,
n
)
)
.
(
25
)
or
wherein the target data calculator is configured to calculate the first or the second side gain so that a value of the first or the second side gain is within a range of +/−20% of a value acquired by the following equations for the left side gain or the right side gain:
G
1
(
ω
)
=
∑
n
=
1
N
⁢
G
grid
,
l
,
n
N
weighted
,
or
G
r
(
ω
)
=
∑
n
=
1
N
⁢
G
grid
,
r
,
n
N
weighted
,
wherein ω is a frequency variable, wherein Nweighted is a normalization parameter, wherein
∑
n
=
1
N
⁢
G
grid
,
l
,
n
is a sum over the stored and potentially modified left variance data items for the identified limited spatial sectors, wherein
∑
n
=
1
N
⁢
G
grid
,
r
,
n
is a sum over the stored and potentially modified right variance data items for the identified limited spatial sectors, wherein N is the number of identified limited spatial sectors, wherein Gl (ω) is the left side gain, wherein Gr (ω) is the right side gain, wherein IACC (ω) is the inter-aural or inter-channel coherence cue, wherein
∑
n
=
1
N
⁢
IACC
grid
,
n
⁢
(
ω
)
is a sum over the stored and potentially modified covariance data items for the identified limited spatial sectors, wherein IALD(ω) is the inter-aural or inter-channel level difference cue, wherein IAPD(ω) is the inter-aural or inter-channel phase difference cue, wherein A2l,n is a squared amplitude of a left side transfer function in the identified limited spatial sector, wherein A2r,n is a squared amplitude of a right side transfer function in the identified limited spatial sector, wherein φl,n is a phase of a left side transfer function in the identified limited spatial sector, wherein φl,n is a phase of a right side transfer function in the identified limited spatial sector, or
wherein the inter-aural or inter-channel phase difference cue is a phase extracted from a transfer function point in a center of the limited spatial sector or from an elementary spatial sector in a center of an extent defined by an identified set of elementary spatial sectors.

288 more elements

The apparatus for synthesizing a spatially extended sound source (SESS), comprising:

a storage for storing one or more rendering data items for different limited spatial sectors, wherein the different limited spatial sectors are located in a rendering range for a listener, wherein the one or more rendering data items for a limited spatial sector comprises at least one of a left variance data item, a right variance data item, and a left-right covariance data item related to the left head related function data and the right head related function data;
a sector identification processor for identifying one or more limited spatial sectors for the spatially extended sound source within the rendering range for the listener based on spatially extended sound source data;
a target data calculator for calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data;
and
an audio processor for processing an audio signal representing the spatially extended sound source using the target rendering data,
wherein the sector identification processor is configured to determine a set of elementary spatial sectors as the one or more limited spatial sectors, wherein, for each elementary spatial sector, at least one of the left variance data item, the right variance data item, and the covariance data item is stored.

3 more elements

A method of synthesizing a spatially extended sound source (SESS), comprising:

storing one or more rendering data items for different limited spatial sectors, wherein the different limited spatial sectors are located in a rendering range for a listener, wherein the one or more rendering data items for a limited spatial sector comprises at least one of a left variance data item related to left head related function data, a right variance data item related to right head related function data, and a covariance data item related to the left head related function data and the right head related function data;
identifying one or more limited spatial sectors for the spatially extended sound source within the rendering range for the listener based on spatially extended sound source data;
calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data;
and
processing an audio signal representing the spatially extended sound source using the target rendering data,
wherein the storing comprises storing, for each limited spatial sector, a frequency dependent representation of the left variance data item, a frequency dependent representation of the right variance data item, and a frequency dependent representation of the covariance data item,
or
wherein the calculating comprises calculating the inter-aural or inter-channel coherence cue, so that a value of the inter-aural or inter-channel coherence cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel coherence cue
IACC
⁡
(
ω
)
=
∑
n
=
1
N
⁢
IACC
grid
,
n
(
ω
)
N
weighted
,
or
wherein the calculating comprises calculating the inter-aural or inter-channel level difference cue so that a value of the inter-aural or inter-channel level difference cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel level difference cue, or
wherein the calculating comprises calculating the inter-aural or inter-channel phase difference cue so that a value of the inter-aural or inter-channel phase difference cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel phase difference cue,
IALD
⁡
(
ω
)
=
10
⁢
log
10
⁢
E
⁢
{
❘
"\[LeftBracketingBar]"
Y
l
(
ω
)
❘
"\[RightBracketingBar]"
2
}
E
⁢
{
❘
"\[LeftBracketingBar]"
Y
r
(
ω
)
❘
"\[RightBracketingBar]"
2
}
=
10
⁢
log
10
⁢
∑
n
=
1
N
⁢
A
l
,
n
2
∑
n
=
1
N
⁢
A
r
,
n
2
,
(
24
)
IAPD
⁡
(
ω
)
=
∠
⁡
(
E
⁢
{
Y
l
(
ω
)
·
Y
r
*
(
ω
)
}
)
=
∠
⁡
(
IACC
⁡
(
ω
)
)
=
∠
⁡
(
∑
n
=
1
N
A
l
,
n
·
A
r
,
n
·
e
j
⁡
(
ϕ
l
,
n
-
ϕ
r
,
n
)
)
.
(
25
)
or
wherein the calculating comprises calculating the first or the second side gain so that a value of the first or the second side gain is within a range of +/−20% of a value acquired by the following equations for the left side gain or the right side gain:
G
l
(
ω
)
=
∑
n
=
1
N
⁢
G
grid
,
l
,
n
N
weighted
,
or
G
r
(
ω
)
=
∑
n
=
1
N
⁢
G
grid
,
r
,
n
N
weighted
,
wherein ω is a frequency variable, wherein Nweighted is a normalization parameter, wherein
∑
n
=
1
N
⁢
G
grid
,
l
,
n
is a sum over the stored and potentially modified left variance data items for the identified limited spatial sectors, wherein
∑
n
=
1
N
⁢
G
grid
,
r
,
n
is a sum over the stored and potentially modified right variance data items for the identified limited spatial sectors, wherein N is the number of identified limited spatial sectors, wherein Gl (ω) is the left side gain, wherein Gr (ω) is the right side gain, wherein IACC (ω) is the inter-aural or inter-channel coherence cue, wherein
∑
n
=
1
N
⁢
IACC
grid
,
n
⁢
(
ω
)
is a sum over the stored and potentially modified covariance data items for the identified limited spatial sectors, wherein IALD(ω) is the inter-aural or inter-channel level difference cue, wherein IAPD(ω) is the inter-aural or inter-channel phase difference cue, wherein A2l,n is a squared amplitude of a left side transfer function in the identified limited spatial sector, wherein A2r,n is a squared amplitude of a right side transfer function in the identified limited spatial sector, wherein φl,m is a phase of a left side transfer function in the identified limited spatial sector, wherein φl,n is a phase of a right side transfer function in the identified limited spatial sector, or
wherein the inter-aural or inter-channel phase difference cue is a phase extracted from a transfer function point in a center of the limited spatial sector or from an elementary spatial sector in a center of an extent defined by an identified set of elementary spatial sectors,
or
wherein the identifying comprises applying a projection algorithm or a ray tracing analysis to determine the one or more limited spatial sectors as a set of elementary spatial sectors, or using, as listener data, a listener position or a listener orientation, or using, as spatially extended sound source (SESS) data, an SESS orientation, an SESS position, or information on a geometry of the SESS,
or
wherein the rendering range comprises a sphere or a portion of a sphere around the listener, wherein the rendering range is tied to the listener position or the listener orientation, wherein the one or more limited spatial sectors have an azimuth size and an elevation size, and wherein the azimuth size and the elevation size of the different limited spatial sectors are different from each other, so that an azimuth size is finer for a limited spatial sector directly in front of the listener compared to an azimuth size of a limited spatial sector more to the side of the listener, or wherein the azimuth size decreases towards a side of the listener, or wherein an elevation size of a limited spatial sector is smaller than an azimuth size of this sector,
or
wherein the identifying comprises determining a set of elementary spatial sectors as the one or more limited spatial sectors, wherein, for each elementary spatial sector, at least one of the left variance data item, the right variance data item, and the covariance data item is stored,
or
wherein the method comprises determining at least one of the left variance data item, the right variance data item, and the covariance data item from pre-stored head related function data, using calculating the left variance data item, the right variance data item or the covariance data item from a plurality of head related function data for the limited spatial sector, and wherein the limited spatial sector is sized in such a way that at least two left head related function data, and at least two right head related function data exist for the limited spatial range.

271 more elements

A non-transitory digital storage medium having a computer program stored thereon to perform, when said computer program is run by a computer, the method of synthesizing a spatially extended sound source (SESS), the method comprising:

storing one or more rendering data items for different limited spatial sectors, wherein the different limited spatial sectors are located in a rendering range for a listener, wherein the one or more rendering data items for a limited spatial sector comprises at least one of a left variance data item related to left head related function data, a right variance data item related to right head related function data, and a covariance data item related to the left head related function data and the right head related function data;
identifying one or more limited spatial sectors for the spatially extended sound source within the rendering range for the listener based on spatially extended sound source data;
calculating target rendering data from the stored left variance data, the stored right variance data, or the stored covariance data;
and
processing an audio signal representing the spatially extended sound source using the target rendering data,
wherein the storing comprises storing, for each limited spatial sector, a frequency dependent representation of the left variance data item, a frequency dependent representation of the right variance data item, and a frequency dependent representation of the covariance data item,
or
wherein the calculating comprises calculating the inter-aural or inter-channel coherence cue, so that a value of the inter-aural or inter-channel coherence cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel coherence cue
IACC
⁡
(
ω
)
=
∑
n
=
1
N
⁢
IACC
grid
,
n
(
ω
)
N
weighted
,
or
wherein the calculating comprises calculating the inter-aural or inter-channel level difference cue so that a value of the inter-aural or inter-channel level difference cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel level difference cue, or
wherein the calculating comprises calculating the inter-aural or inter-channel phase difference cue so that a value of the inter-aural or inter-channel phase difference cue is within a range of +/−20% of a value acquired by the following equation for the inter-aural or inter-channel phase difference cue,
IALD
⁡
(
ω
)
=
10
⁢
log
10
⁢
E
⁢
{
❘
"\[LeftBracketingBar]"
Y
l
(
ω
)
❘
"\[RightBracketingBar]"
2
}
E
⁢
{
❘
"\[LeftBracketingBar]"
Y
r
(
ω
)
❘
"\[RightBracketingBar]"
2
}
=
10
⁢
log
10
⁢
∑
n
=
1
N
⁢
A
l
,
n
2
∑
n
=
1
N
⁢
A
r
,
n
2
,
(
24
)
IAPD
⁡
(
ω
)
=
∠
⁡
(
E
⁢
{
Y
l
(
ω
)
·
Y
r
*
(
ω
)
}
)
=
∠
⁡
(
IACC
⁡
(
ω
)
)
=
∠
⁡
(
∑
n
=
1
N
A
l
,
n
·
A
r
,
n
·
e
j
⁡
(
ϕ
l
,
n
-
ϕ
r
,
n
)
)
.
(
25
)
or
wherein the calculating comprises calculating the first or the second side gain so that a value of the first or the second side gain is within a range of +/−20% of a value acquired by the following equations for the left side gain or the right side gain:
G
l
(
ω
)
=
∑
n
=
1
N
⁢
G
grid
,
l
,
n
N
weighted
,
or
G
r
(
ω
)
=
∑
n
=
1
N
⁢
G
grid
,
r
,
n
N
weighted
,
wherein w is a frequency variable, wherein Nweighted is a normalization parameter, wherein
∑
n
=
1
N
⁢
G
grid
,
l
,
n
is a sum over the stored and potentially modified left variance data items for the identified limited spatial sectors, wherein
∑
n
=
1
N
⁢
G
grid
,
r
,
n
is a sum over the stored and potentially modified right variance data items for the identified limited spatial sectors, wherein N is the number of identified limited spatial sectors, wherein Gl (ω) is the left side gain, wherein Gr (ω) is the right side gain, wherein IACC (ω) is the inter-aural or inter-channel coherence cue, wherein
∑
n
=
1
N
⁢
IACC
grid
,
n
⁢
(
ω
)
is a sum over the stored and potentially modified covariance data items for the identified limited spatial sectors, wherein IALD(ω) is the inter-aural or inter-channel level difference cue, wherein IAPD(ω) is the inter-aural or inter-channel phase difference cue, wherein A2l,n is a squared amplitude of a left side transfer function in the identified limited spatial sector, wherein A2r,n is a squared amplitude of a right side transfer function in the identified limited spatial sector, wherein φl,n is a phase of a left side transfer function in the identified limited spatial sector, wherein φl,n is a phase of a right side transfer function in the identified limited spatial sector, or
wherein the inter-aural or inter-channel phase difference cue is a phase extracted from a transfer function point in a center of the limited spatial sector or from an elementary spatial sector in a center of an extent defined by an identified set of elementary spatial sectors,
or
wherein the identifying comprises applying a projection algorithm or a ray tracing analysis to determine the one or more limited spatial sectors as a set of elementary spatial sectors, or using, as listener data, a listener position or a listener orientation, or using, as spatially extended sound source (SESS) data, an SESS orientation, an SESS position, or information on a geometry of the SESS,
or
wherein the rendering range comprises a sphere or a portion of a sphere around the listener, wherein the rendering range is tied to the listener position or the listener orientation, wherein the one or more limited spatial sectors have an azimuth size and an elevation size, and wherein the azimuth size and the elevation size of the different limited spatial sectors are different from each other, so that an azimuth size is finer for a limited spatial sector directly in front of the listener compared to an azimuth size of a limited spatial sector more to the side of the listener, or wherein the azimuth size decreases towards a side of the listener, or wherein an elevation size of a limited spatial sector is smaller than an azimuth size of this sector,
or
wherein the identifying comprises determining a set of elementary spatial sectors as the one or more limited spatial sectors, wherein, for each elementary spatial sector, at least one of the left variance data item, the right variance data item, and the covariance data item is stored,
or
wherein the method comprises determining at least one of the left variance data item, the right variance data item, and the covariance data item from pre-stored head related function data, using calculating the left variance data item, the right variance data item or the covariance data item from a plurality of head related function data for the limited spatial sector, and wherein the limited spatial sector is sized in such a way that at least two left head related function data, and at least two right head related function data exist for the limited spatial range.

271 more elements

Dependencies are inferred from the claim text.

Apparatus, method or computer program for synthesizing a spatially extended sound source using variance or covariance data

Life of this patent

Apparatus, method or computer program for synthesizing a spatially extended sound source using variance or covariance data

Life of this patent

Abstract

Claims21