# US 8,280,744 B2 · Audio decoder, audio object encoder, method for decoding a multi-audio-object signal, multi-audio-object encoding method, and non-transitory computer-readable medium therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 8,280,744 B2
- Document ID: US8280744B2
- Status: Granted
- Kind: B2
- Application number: 12/253,515
- Filed: 2008-10-17
- Published: 2009-05-14
- Granted: 2012-10-02
- Earliest claimed: 2007-10-17
- Assignees: Fraunhofer-Gesellschaft zur Foerderung der angewandten Forschung e.V.
- Inventors: Oliver Hellmuth; Johannes Hilpert; Leonid Terentiev; Cornelia Falch; Andreas Hoelzer; Juergen Herre
- Web page: https://patentgrep.com/patent/US8280744B2

## CPC

- G10L 19/04: using predictive techniques
- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- H04S 3/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- G10L 19/20: using sound class specific coding, hybrid encoders or object based coding
- H04S 2420/03: Application of parametric coding in stereophonic audio systems
- H04S 2420/07: Synergistic effects of band splitting and sub-band processing
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04S: STEREOPHONIC SYSTEMS

## Abstract

An audio decoder for decoding a multi-audio-object signal having an audio signal of a first type and an audio signal of a second type encoded therein is described, the multi-audio-object signal having a downmix signal and side information, the side information having level information of the audio signals of the first and second types in a first predetermined time/frequency resolution, and a residual signal specifying residual level values in a second predetermined time/frequency resolution, the audio decoder having a processor for computing prediction coefficients based on the level information; and an up-mixer for up-mixing the downmix signal based on the prediction coefficients and the residual signal to obtain a first up-mix audio signal approximating the audio signal of the first type and/or a second up-mix audio signal approximating the audio signal of the second type.

## Claims

### Claim 1

An audio decoder for decoding a multi-audio-object signal comprising an audio signal of a first type and an audio signal of a second type encoded therein, the multi-audio-object signal comprising a downmix signal and side information, the side information comprising level information of the audio signal of the first type and the audio signal of the second type in a first predetermined time/frequency resolution, and a residual signal specifying residual level values in a second predetermined time/frequency resolution, the audio decoder comprising
a processor for computing prediction coefficients based on the level information; and
an up-mixer for up-mixing the downmix signal based on the prediction coefficients and the residual signal to acquire a first up-mix audio signal approximating the audio signal of the first type and/or a second up-mix audio signal approximating the audio signal of the second type; wherein
the processor for computing prediction coefficients based on the level information is configured to compute channel prediction coefficients cil,m for each time/frequency tile (l,m) of the first (l,m) predetermined time/frequency resolution, for each output channel i of the downmix signal as
c
1
l
,
m
=
P
LoFo
l
,
m
⁢
P
Ro
l
,
m
-
P
RoFo
l
,
m
⁢
P
LoRo
l
,
m
P
Lo
l
,
m
⁢
P
Ro
l
,
m
-
P
LoRo
2
⁢
⁢
l
,
m
⁢
⁢
and
⁢
⁢
c
2
l
,
m
=
P
RoFo
l
,
m
⁢
P
Lo
l
,
m
-
P
LoFo
l
,
m
⁢
P
LoRo
l
,
m
P
Lo
l
,
m
⁢
P
Ro
l
,
m
-
P
LoRo
2
⁢
⁢
l
,
m
with
P
Lo
=
OLD
L
+
m
F
2
⁢
OLD
F
,
⁢
P
Ro
=
OLD
R
+
n
F
2
⁢
OLD
F
,
⁢
P
LoRo
=
IOC
LR
⁢
OLD
L
⁢
OLD
R
+
m
F
⁢
n
F
⁢
OLD
F
,
⁢
P
LoF
=
m
F
⁢
OLD
L
+
n
F
⁢
IOC
LR
⁢
OLD
L
⁢
OLD
R
-
m
F
⁢
OLD
F
,
⁢
P
RoF
=
n
F
⁢
OLD
R
+
m
F
⁢
IOC
LR
⁢
OLD
L
⁢
OLD
R
-
n
F
⁢
OLD
F
,
with OLDL denoting a normalized spectral energy of a first input channel of the audio signal of the first type at the respective time/frequency tile, OLDR denoting the normalized spectral energy of a second input channel of the audio signal of the first type at the respective time/frequency tile, and IOCLR denoting inter-correlation information defining spectral energy similarity between the first and second input channel of the audio signal of the first type within the respective time/frequency tile—in case the audio signal of the first type is stereo—, or OLDL denoting the normalized spectral energy of the audio signal of the first type at the respective time/frequency tile, and OLDR and IOCLR being zero—in case same is mono,
and with OLDF denoting the normalized spectral energy of the audio signal of the second type at the respective time/frequency tile,
with
m
F
=
10
0.05
⁢
⁢
DMG
F
⁢
10
0.1
⁢
⁢
DCLD
F
1
+
10
0.1
⁢
⁢
DCLD
F
⁢
⁢
and
n
F
=
10
0.05
⁢
⁢
DMG
F
⁢
1
1
+
10
0.1
⁢
⁢
DCLD
F
,
where DCLDF and DMGF are downmix prescriptions contained in the side information; and
the up-mixer is configured to yield the first up-mix signal S1 and/or the second up-mix signal S2 from the downmix signal d and a residual signal res via
(
S
1
S
2
)
=
D
-
1
⁡
(
1
0
C
1
)
⁢
(
d
n
,
k
res
n
,
k
)
,
where the “1” in the top left-hand corner denotes—depending on the number of channels of dn,k—a scalar, or an identity matrix, C is—depending on the number of channels of dn,k—c1n,k or
(
c
1
n
,
k
c
2
n
,
k
)
T
,
the “1” in the bottom right-hand corner is a scalar, “0” denotes—depending on the number of channels of dn,k—a zero vector or a scalar and D−1 is a matrix uniquely determined by a downmix prescription according to which the audio signal of the first type and the audio signal of the second type are downmixed into the downmix signal, and which is also comprised by the side information, and dn,k and resn,k denote the downmix signal and the residual signal at time/frequency tile (n,k), respectively.

### Claim 2 (depends on 1)

The audio decoder according to claim 1, wherein the side information further comprises a downmix prescription according to which the audio signal of the first type and the audio signal of the second type are downmixed into the downmix signal, wherein the up-mixer is configured to perform the up-mixing further based on the downmix prescription.

### Claim 3 (depends on 2)

The audio decoder according to claim 2, wherein the downmix prescription varies in time within the side information.

### Claim 4 (depends on 2)

The audio decoder according to claim 2, wherein the downmix prescription varies in time within the side information at a time resolution coarser than a frame-size.

### Claim 5 (depends on 2)

The audio decoder according to claim 2, wherein the downmix prescription indicates the weighting by which the downmix signal has been mixed-up based on the audio signal of the first type and the audio signal of the second type.

### Claim 6 (depends on 1)

The audio decoder according to claim 1, wherein the audio signal of the first type is a stereo audio signal comprising a first and a second input channel, or a mono audio signal comprising only a first input channel, and the downmix signal is a stereo audio signal comprising a first and second output channel, or a mono audio signal comprising only a first output channel wherein the level information describes level differences between the first input channel, the second input channel and the audio signal of the second type, respectively, at the first predetermined time/frequency resolution, wherein the side information further comprises inter-correlation information defining level similarities between the first and second input channel in a third predetermined time/frequency resolution, wherein the processor is configured to perform the computation further based on the inter-correlation information.

### Claim 7 (depends on 6)

The audio decoder according to claim 6, wherein the first and third time/frequency resolutions are determined by a common syntax element within the side information.

### Claim 8 (depends on 6)

The audio decoder according to claim 6, wherein the processor and the up-mixer are configured such that the up-mixing is representable by an appliance of a vector composed of the downmix signal and the residual signal, to a sequence of a first and a second matrix, the first matrix being composed of the prediction coefficients and the second matrix being defined by a downmix prescription according to which the audio signal of the first type and the audio signal of the second type are downmixed into the downmix signal, and which is also comprised by the side information.

### Claim 9 (depends on 8)

The audio decoder according to claim 8, wherein the processor and the up-mixer are configured such that the first matrix maps the vector to an intermediate vector comprising a first component for the audio signal of the first type and/or a second component for the audio signal of the second type and being defined such that the downmix signal is mapped onto the first component 1-to-1, and a linear combination of the residual signal and the downmix signal is mapped onto the second component.

### Claim 10 (depends on 1)

The audio decoder according to claim 1, wherein the multi-audio-object signal comprises a plurality of audio signals of the second type and the side information comprises one residual signal per audio signal of the second type.

### Claim 11 (depends on 1)

The audio decoder according to claim 1, wherein the second predetermined time/frequency resolution is related to the first predetermined time/frequency resolution via a residual resolution parameter comprised in the side information, wherein the audio decoder is configured to derive the residual resolution parameter from the side information.

### Claim 12 (depends on 11)

The audio decoder according to claim 11, wherein the residual resolution parameter defines a spectral range over which the residual signal is transmitted within the side information.

### Claim 13 (depends on 12)

The audio decoder according to claim 12, wherein the residual resolution parameter defines a lower and an upper limit of the spectral range.

### Claim 14 (depends on 1)

The audio decoder according to claim 1, wherein D−1 is the inversion of
D
=
(
1
0
m
F
0
1
n
F
m
F
n
F
-
1
)
in case of the downmix signal being stereo and S1 being stereo,
D
=
(
1
m
F
1
n
F
m
F
+
n
F
-
1
)
in case of the downmix signal being stereo and S1 being mono,
D
=
(
1
1
m
F
m
F
/
2
m
F
/
2
-
1
)
in case of the downmix signal being mono and S1 being stereo, or
D
=
(
1
m
F
m
F
-
1
)
in case of the downmix signal being mono and S1 being mono.

### Claim 15 (depends on 1)

The audio decoder according to claim 1, wherein the multi-audio-object signal comprises spatial rendering information for spatially rendering the audio signal of the first type onto a predetermined loudspeaker configuration.

### Claim 16 (depends on 1)

The audio decoder according to claim 1, wherein the upmixer is configured to spatially render the first up-mix audio signal separated from the second up-mix audio signal, spatially render the second up-mix audio signal separated from the first up-mix audio signal, or mix the first up-mix audio signal and the second up-mix audio signal and spatially render the mixed version thereof onto a predetermined loudspeaker configuration.

### Claim 17

A method for decoding a multi-audio-object signal comprising an audio signal of a first type and an audio signal of a second type encoded therein, the multi-audio-object signal comprising a downmix signal and side information, the side information comprising level information of the audio signal of the first type and the audio signal of the second type in a first predetermined time/frequency resolution, and a residual signal specifying residual level values in a second predetermined time/frequency resolution, the method comprising
computing prediction coefficients based on the level information; and
up-mixing the downmix signal based on the prediction coefficients and the residual signal to acquire a first up-mix audio signal approximating the audio signal of the first type and/or a second up-mix audio signal approximating the audio signal of the second type; wherein
the computing the prediction coefficients based on the level information comprises computing channel prediction coefficients cil,m for each time/frequency tile (l,m) of the first (l,m) predetermined time/frequency resolution, for each output channel i of the downmix signal as
c
1
l
,
m
=
P
LoFo
l
,
m
⁢
P
Ro
l
,
m
-
P
RoFo
l
,
m
⁢
P
LoRo
l
,
m
P
Lo
l
,
m
⁢
P
Ro
l
,
m
-
P
LoRo
2
⁢
⁢
l
,
m
⁢
⁢
and
⁢
⁢
c
2
l
,
m
=
P
RoFo
l
,
m
⁢
P
Lo
l
,
m
-
P
LoFo
l
,
m
⁢
P
LoRo
l
,
m
P
Lo
l
,
m
⁢
P
Ro
l
,
m
-
P
LoRo
2
⁢
⁢
l
,
m
with
P
Lo
=
OLD
L
+
m
F
2
⁢
OLD
F
,
⁢
P
Ro
=
OLD
R
+
n
F
2
⁢
OLD
F
,
⁢
P
LoRo
=
IOC
LR
⁢
OLD
L
⁢
OLD
R
+
m
F
⁢
n
F
⁢
OLD
F
,
⁢
P
LoF
=
m
F
⁢
OLD
L
+
n
F
⁢
IOC
LR
⁢
OLD
L
⁢
OLD
R
-
m
F
⁢
OLD
F
,
⁢
P
RoF
=
n
F
⁢
OLD
R
+
m
F
⁢
IOC
LR
⁢
OLD
L
⁢
OLD
R
-
n
F
⁢
OLD
F
,
with OLDL denoting a normalized spectral energy of a first input channel of the audio signal of the first type at the respective time/frequency tile, OLDR denoting the normalized spectral energy of a second input channel of the audio signal of the first type at the respective time/frequency tile, and IOCLR denoting inter-correlation information defining spectral energy similarity between the first and second input channel of the audio signal of the first type within the respective time/frequency tile—in case the audio signal of the first type is stereo —, or OLDL denoting the normalized spectral energy of the audio signal of the first type at the respective time/frequency tile, and OLDR and IOCLR being zero—in case same is mono,
and with OLDF denoting the normalized spectral energy of the audio signal of the second type at the respective time/frequency tile,
with
m
F
=
10
0.05
⁢
⁢
DMG
F
⁢
10
0.1
⁢
⁢
DCLD
F
1
+
10
0.1
⁢
⁢
DCLD
F
⁢
⁢
and
n
F
=
10
0.05
⁢
⁢
DMG
F
⁢
1
1
+
10
0.1
⁢
⁢
DCLD
F
,
where DCLDF and DMGF are downmix prescriptions contained in the side information; and
the up-mixing comprises yielding the first up-mix signal S1 and/or the second up-mix signal S2 from the downmix signal d and a residual signal res via
(
S
1
S
2
)
=
D
-
1
⁡
(
1
0
C
1
)
⁢
(
d
n
,
k
res
n
,
k
)
,
where the “1” in the top left-hand corner denotes—depending on the number of channels of dn,k—a scalar, or an identity matrix, C is—depending on the number of channels of dn,k—c1n,k or
(
c
1
n
,
k
c
2
n
,
k
)
T
,
the “1” in the bottom right-hand corner is a scalar, “0” denotes—depending on the number of channels of dn,k—a zero vector or a scalar and D−1 is a matrix uniquely determined by a downmix prescription according to which the audio signal of the first type and the audio signal of the second type are downmixed into the downmix signal, and which is also comprised by the side information, and dn,k and resn,k denote the downmix signal and the residual signal at time/frequency tile (n,k), respectively.

### Claim 18

A non-transitory computer-readable medium having stored thereon a computer program with a program code for executing, when running on a processor, a method for decoding a multi-audio-object signal comprising an audio signal of a first type and an audio signal of a second type encoded therein, the multi-audio-object signal comprising a downmix signal and side information, the side information comprising level information of the audio signal of the first type and the audio signal of the second type in a first predetermined time/frequency resolution, and a residual signal specifying residual level values in a second predetermined time/frequency resolution, the method comprising
computing prediction coefficients based on the level information; and
up-mixing the downmix signal based on the prediction coefficients and the residual signal to acquire a first up-mix audio signal approximating the audio signal of the first type and/or a second up-mix audio signal approximating the audio signal of the second type; wherein
the computing the prediction coefficients based on the level information comprises computing channel prediction coefficients cil,m for each time/frequency tile (l,m) of the first (l,m) predetermined time/frequency resolution, for each output channel i of the downmix signal as
c
1
l
,
m
=
P
LoFo
l
,
m
⁢
P
Ro
l
,
m
-
P
RoFo
l
,
m
⁢
P
LoRo
l
,
m
P
Lo
l
,
m
⁢
P
Ro
l
,
m
-
P
LoRo
2
⁢
⁢
l
,
m
⁢
⁢
and
⁢
⁢
c
2
l
,
m
=
P
RoFo
l
,
m
⁢
P
Lo
l
,
m
-
P
LoFo
l
,
m
⁢
P
LoRo
l
,
m
P
Lo
l
,
m
⁢
P
Ro
l
,
m
-
P
LoRo
2
⁢
⁢
l
,
m
with
P
Lo
=
OLD
L
+
m
F
2
⁢
OLD
F
,
⁢
P
Ro
=
OLD
R
+
n
F
2
⁢
OLD
F
,
⁢
P
LoRo
=
IOC
LR
⁢
OLD
L
⁢
OLD
R
+
m
F
⁢
n
F
⁢
OLD
F
,
⁢
P
LoF
=
m
F
⁢
OLD
L
+
n
F
⁢
IOC
LR
⁢
OLD
L
⁢
OLD
R
-
m
F
⁢
OLD
F
,
⁢
P
RoF
=
n
F
⁢
OLD
R
+
m
F
⁢
IOC
LR
⁢
OLD
L
⁢
OLD
R
-
n
F
⁢
OLD
F
,
with OLDL denoting a normalized spectral energy of a first input channel of the audio signal of the first type at the respective time/frequency tile, OLDR denoting the normalized spectral energy of a second input channel of the audio signal of the first type at the respective time/frequency tile, and IOCLR denoting inter-correlation information defining spectral energy similarity between the first and second input channel of the audio signal of the first type within the respective time/frequency tile—in case the audio signal of the first type is stereo —, or OLDL denoting the normalized spectral energy of the audio signal of the first type at the respective time/frequency tile, and OLDR and IOCLR being zero—in case same is mono,
and with OLDF denoting the normalized spectral energy of the audio signal of the second type at the respective time/frequency tile,
with
m
F
=
10
0.05
⁢
⁢
DMG
F
⁢
10
0.1
⁢
⁢
DCLD
F
1
+
10
0.1
⁢
⁢
DCLD
F
⁢
⁢
and
n
F
=
10
0.05
⁢
⁢
DMG
F
⁢
1
1
+
10
0.1
⁢
⁢
DCLD
F
,
where DCLDF and DMGF are downmix prescriptions contained in the side information; and
the up-mixing comprises yielding the first up-mix signal S1 and/or the second up-mix signal S2 from the downmix signal d and a residual signal res via
(
S
1
S
2
)
=
D
-
1
⁡
(
1
0
C
1
)
⁢
(
d
n
,
k
res
n
,
k
)
,
where the “1” in the top left-hand corner denotes—depending on the number of channels of dn,k—a scalar, or an identity matrix, C is—depending on the number of channels of dn,k—c1n,k or
(
c
1
n
,
k
c
2
n
,
k
)
T
,
the “1” in the bottom right-hand corner is a scalar, “0” denotes—depending on the number of channels of dn,k—a zero vector or a scalar and D−1 is a matrix uniquely determined by a downmix prescription according to which the audio signal of the first type and the audio signal of the second type are downmixed into the downmix signal, and which is also comprised by the side information, and dn,k and resn,k denote the downmix signal and the residual signal at time/frequency tile (n,k), respectively.
