# US 2022/0036911 A1 · Apparatus, method or computer program for generating an output downmix representation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2022/0036911 A1
- Document ID: US20220036911A1
- Status: Published application
- Kind: A1
- Application number: 17/501,993
- Filed: 2021-10-14
- Published: 2022-02-03
- Earliest claimed: 2019-04-23
- Assignees: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
- Applicants: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
- Inventors: Franz REUTELHUBER; Eleni FOTOPOULOU; Markus MULTRUS
- Web page: https://patentgrep.com/patent/US20220036911A1

## CPC

- G10L 21/04: Time compression or expansion
- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- G10L 19/022: Blocking, i.e. grouping of samples in time; Choice of analysis windows; Overlap factoring
- H04S 1/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- H04S 1/007: {in which the audio signals are in digital form (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- H04S 3/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- H04S 7/30: {Control circuits for electronic adaptation of the sound field}
- H04S 2400/01: Multi-channel, i.e. more than two input channels, sound reproduction with two speakers wherein the multi-channel information is substantially preserved
- H04S 2400/03: Aspects of down-mixing multi-channel audio to configurations with lower numbers of playback channels, e.g. 7.1 -> 5.1 (H04S2400/01 takes precedence)
- H04S 2400/05: Generation or adaptation of centre channel in multi-channel audio systems
- H04S 2420/07: Synergistic effects of band splitting and sub-band processing
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04S: STEREOPHONIC SYSTEMS

## Abstract

An apparatus for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, includes: an upmixer for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to obtain at least one upmixed portion; and a downmixer for downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme.

## Claims

### Claim 1

Apparatus for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the apparatus comprising:
an upmixer for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire at least one upmixed portion; and
a downmixer for downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation.

### Claim 2 (depends on 1)

Apparatus of claim 1, wherein only the portion of the input downmix representation is in accordance with the first downmixing scheme and a second portion of the input downmix representation is in accordance with the second downmixing scheme,
wherein the downmixer is configured for downmixing the at least one upmixed portion in accordance with the second downmixing scheme to acquire the first downmixed portion; and
further comprising a combiner for combining the first downmixed portion and the second portion of the input downmix representation or a downmixed portion derived from the second portion of the input downmix representation to acquire the output downmix representation comprising a first output representation for only the portion of the input downmix representation and a second output representation for the second portion of the input downmix representation, wherein the first output representation for only the portion of the input downmix representation and the second output representation for the second portion of the input downmix representation are based on the same downmixing scheme.

### Claim 3 (depends on 1)

Apparatus of claim 1,
wherein the at least the portion of the input downmix representation or only the portion of the input downmix representation is a first frequency band, wherein the first downmixing scheme is a downmixing scheme relying on a residual signal, and
wherein the upmixer is configured to perform an upmix using the residual signal.

### Claim 4 (depends on 1)

Apparatus of claim 1, wherein the second downmixing scheme is a fully parametric scheme, and wherein the downmixer is configured to apply the second downmixing scheme.

### Claim 5 (depends on 2)

Apparatus of claim 2, wherein the second portion of the input downmix representation is a second frequency band, and wherein the combiner is configured to combine the first downmixed portion and the second portion of the input downmix representation to acquire the output downmix representation.

### Claim 6 (depends on 1)

Apparatus of claim 1, further comprising an audio decoder for generating a decoded core signal for at least the portion of the input downmix representation or only the portion of the input downmix representation, and a decoded residual signal for at least the portion of the input downmix representation or only the portion of the input downmix representation,
wherein the upmixer is configured to use, in the upmixing scheme, the decoded core signal for at least the portion of the input downmix representation or only the portion of the input downmix representation and the decoded residual signal for at least the portion of the input downmix representation or only the portion of the input downmix representation,
wherein the downmixer is configured for receiving the at least one upmixed portion comprising more channels than the input downmix representation.

### Claim 7 (depends on 6)

Apparatus of claim 6, wherein the second portion of the input downmix representation is in accordance with the second downmixing scheme, wherein the audio decoder is configured for generating a decoded core signal for the second portion of the input downmix representation and a decoded residual signal for at least the portion of the input downmix representation or only the portion of the input downmix representation only, and wherein the combiner is configured to combine the first downmixed portion and the decoded core signal for the second portion of the input downmix representation.

### Claim 8 (depends on 1)

Apparatus of claim 1, further comprising:
a time-to-spectrum converter for converting a time domain input downmix representation of at least the portion of the input downmix representation or only the portion of the input downmix representation into a spectral domain; and a spectrum-to-time converter for converting an output signal into a time domain to acquire the output downmix representation, wherein the time-to-spectrum converter or the spectrum-to-time converter is configured to perform an overlap and add processing or to perform a crossover processing from an earlier time block to a later time block, or
further comprising an output interface for outputting the output downmix representation to a rendering device or further comprising a rendering device for rendering the output downmix representation as a mono replay signal, or
wherein the downmixer is configured to apply, as the second downmixing scheme, an active downmixing scheme, an energy conserving downmixing scheme, or a downmixing scheme, in which a target energy of the downmix signal is in a predetermined ratio to an energy of a mid-channel derived from a first channel and a second channel, wherein at least one of the first channel and the second channel is phase rotated before being added together to form the input downmix representation.

### Claim 9 (depends on 8)

Apparatus of claim 8, wherein the second portion of the input downmix representation is in accordance with the second downmixing, wherein the time-to-spectrum converter is configured for converting a time domain input downmix representation of the second portion of the input downmix representation into the spectral domain, or
wherein the predetermined ratio indicates an equality or a deviation range being 3 dB related to a higher energy of energies of a first original channel and a second original channel.

### Claim 10 (depends on 1)

Apparatus of claim 1, wherein at least the portion of the input downmix representation is in accordance with the first downmixing scheme relying on a residual signal or on a residual signal and parametric information,
wherein the upmixer is configured for upmixing the input downmix representation of at least the portion of the input downmix representation using the upmixing scheme corresponding to the first downmixing scheme and using the residual signal or the residual signal and the parametric information, respectively to acquire the at least one upmixed portion; and
wherein the downmixer is configured for downmixing the at least one upmixed portion in accordance with the second downmixing scheme different from the first downmixing scheme, wherein the second downmixing scheme is an active downmixing scheme or a fully parametric downmixing scheme to acquire the output downmix representation comprising at least one downmixed portion.

### Claim 11 (depends on 10)

Apparatus of claim 10, further comprising an output interface for outputting the output downmix representation to a rendering device or further comprising a rendering device for rendering the output downmix representation as a mono replay signal.

### Claim 12 (depends on 10)

Apparatus of claim 10, wherein the downmixer is configured to apply, as the active downmixing scheme, an energy conserving downmixing scheme, or a downmixing scheme, in which a target energy of the downmix signal is in a predetermined ratio to an energy of a mid-channel derived from a first channel and a second channel, wherein at least one of the first channel and the second channel is phase rotated before being added together.

### Claim 13 (depends on 10)

Apparatus of claim 10, wherein at least the portion of the input downmix representation comprises the full bandwidth of the input downmix representation.

### Claim 14 (depends on 1)

Apparatus of claim 1,
wherein the downmixer is configured to perform the second downmixing scheme, the second downmixing scheme comprising:
calculating a first weight for a first channel and a second weight for a second channel for a spectral band of the at least one upmixed portion, the spectral band comprising a plurality of spectral lines, and
applying the first weight to spectral lines of the spectral band of the first channel and applying the second weight to spectral lines of the spectral band of the second channel, and adding first weighted lines and second weighted lines to acquire downmixed spectral lines in the spectral band, and
wherein the apparatus is configured to convert the downmixed spectral lines to a time domain to acquire time domain samples of the output downmix representation.

### Claim 15 (depends on 14)

Apparatus of claim 14, wherein the calculation of the first weight and the second weight is performed band wise using energies of the first channel and the second channel and a target energy.

### Claim 16 (depends on 15)

Apparatus of claim 15, wherein the target energy is equal to an energy of a phase-rotated mid-channel or is derived from the energies of the first channel, the second channel and from a correlation value between the first channel and the second channel.

### Claim 17 (depends on 14)

Apparatus of claim 14, wherein calculating the first weight and the second weight comprises, for a spectral band:
calculating an amplitude-related measure for the first channel in the spectral band;
calculating an amplitude-related measure for the second channel in the spectral band:
calculating an amplitude-related measure for a linear combination of the first channel and the second channel in the spectral band;
calculating a cross-correlation measure between the first channel and the second channel in the spectral band; and
calculating the first weight and the second weight using the amplitude-related measure for the first channel, the amplitude-related measure for the second channel, the amplitude-related measure for the linear combination and the cross-correlation measure.

### Claim 18 (depends on 1)

Apparatus of claim 1, wherein the upmixer is configured to perform the upmixing scheme, the upmixing scheme comprising:
calculating first channel spectral lines for a spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation from spectral lines of the spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation using a prediction parameter for the spectral band and residual signal lines for the spectral band and a first calculation rule, and
calculating second channel spectral lines for the spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation from the spectral lines of the spectral band of at least the portion of the input downmix representation or only the portion of the input downmix representation using the prediction parameter for the spectral band and the residual signal lines for the spectral band and a second calculation rule,
wherein the first calculation rule is different from the second calculation rule.

### Claim 19 (depends on 18)

Apparatus of claim 18, wherein the first calculation rule comprises one of an addition and a subtraction and the second calculation rule comprises the other one of the addition and the subtraction.

### Claim 20

Multichannel decoder, comprising:
an input interface for providing an input downmix representation and parametric data at least for a second portion of the input downmix representation; and
the apparatus for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, said apparatus comprising:
an upmixer for upmixing at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire at least one upmixed portion; and
a downmixer for downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation,
wherein the multichannel decoder is configured to upmix, with the upmixer, the input downmix representation for at least the portion of the input downmix representation or only the portion of the input downmix representation in accordance with the upmixing scheme corresponding to the first downmixing scheme to acquire the at least one upmixed portion, and/or to upmix the input downmix representation for the second portion and the parametric data using a second upmixing scheme corresponding to the second downmixing scheme to acquire an upmixed second portion, and
wherein a combiner is configured to combine the at least one upmixed portion and the upmixed second portion to acquire a multichannel output signal.

### Claim 21 (depends on 20)

Multichannel decoder of claim 20, wherein the input interface comprises:
a first time-spectrum converter for converting a first spectral representation of the at least the portion of the input downmix representation or only the portion of the input downmix representation and a second spectral representation of a second portion of the input downmix representation, the second portion of the input downmix representation comprising spectral values for higher frequencies than at least the portion of the input downmix representation or only the portion of the input downmix representation of the first spectral representation;
a second time-spectrum-converter for generating a spectral representation of a residual signal for the at least the portion of the input downmix representation or only the portion of the input downmix representation,
wherein the upmixer is configured to upmix the first spectral representation using the spectral representation of the residual signal to acquire the at least one upmixed portion in the spectral domain,
wherein the downmixer is configured to downmix the at least one upmixed portion to acquire the first downmixed portion in the spectral domain, and
wherein the combiner comprises a spectrum-time converter for combining the first downmixed portion and the spectral representation of the second portion of the input downmix representation and for converting into the time domain to acquire the output downmix representation.

### Claim 22 (depends on 20)

Multichannel decoder of claim 20, further comprising:
a second upmixer for upmixing the second portion of the input downmix representation to acquire the upmixed second portion,
wherein, in a multichannel output mode, the combiner is configured to combine a first channel of the at least one upmixed portion and the first channel of the upmixed second portion and to convert into a time domain to acquire a first channel of a multichannel output,
wherein the multichannel decoder further comprises a second combiner configured to combine, in the multichannel output mode, a second channel of the at least one upmixed portion and a second channel of the upmixed second portion and to convert into the time domain to acquire a second channel of the multichannel output.

### Claim 23 (depends on 21)

Multichannel decoder of claim 21, further comprising:
a second upmixer for upmixing the second portion of the input downmix representation to acquire the upmixed second portion,
wherein, in a multichannel output mode, the combiner is configured to combine a first channel of the at least one upmixed portion and the first channel of the upmixed second portion and to convert into a time domain to acquire a first channel of a multichannel output,
wherein the multichannel decoder further comprises a second combiner configured to combine, in the multichannel output mode, a second channel of the at least one upmixed portion and a second channel of the upmixed second portion and to convert into the time domain to acquire a second channel of the multichannel output,
a switch connected between the first time-spectrum-converter and the second upmixer, and
a controller, wherein the controller is configured to control, in a mono-output mode, the switch to connect an output of the first time-spectrum-converter to the combiner or to bypass the second upmixer and to connect an output of the upmixer to an input of the downmixer, or to control, in the multichannel output mode, the switch to connect an output of the first time-spectrum-converter to an input of the second upmixer.

### Claim 24 (depends on 22)

Multichannel decoder of claim 22, further comprising a second switch connected between the upmixer and the downmixer; and
a controller, wherein the controller is configured to control, in the mono-output mode, the second switch to connect an output of the upmixer to an input of the downmixer and to control, in the multichannel output mode, the second switch to connect an output of the upmixer to an input of the second combiner or to bypass the downmixer.

### Claim 25

Method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the method comprising:
upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and
downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation.

### Claim 26 (depends on 25)

Method of claim 25, wherein a second portion of the input downmix representation is in accordance with a second downmixing scheme,
wherein the downmixing comprises downmixing the at least one upmixed portion in accordance with the second downmixing scheme to acquire the first downmixed portion; and
wherein the method further comprises combining the first downmixed portion and the second portion or a downmixed portion derived from the second portion to acquire the output downmix representation, wherein the output downmix representation for at least the portion of the input downmix representation and the output representation for the second portion are based on the same downmixing scheme.

### Claim 27 (depends on 25)

Method of claim 25, wherein at least the portion of the input downmix representation is in accordance with the first downmixing scheme relying on a residual signal or on a residual signal and parametric information,
wherein the upmixing comprises upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme and using the residual signal or the residual signal and the parametric information, respectively to acquire the at least one upmixed portion; and
wherein the downmixing comprises downmixing the at least one upmixed portion in accordance with the second downmixing scheme different from the first downmixing scheme, wherein the second downmixing scheme is an active downmixing scheme or a fully parametric downmixing scheme to acquire the output downmix representation for at least the portion of the input downmix representation.

### Claim 28

Method of multichannel decoding, comprising:
providing an input downmix representation and parametric data at least for a second portion of the input downmix representation;
the method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the method for generating an output downmix representation comprising:
upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and
downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation,
wherein the method comprises the upmixing the input downmix representation for at least the portion of the input downmix representation or only the portion of the input downmix representation in accordance with the upmixing scheme corresponding to the first downmixing scheme to acquire the at least one upmixed portion, and/or upmixing the second portion of the input downmix representation and the parametric data using a second upmixing scheme corresponding to the second downmixing scheme to acquire an upmixed second portion, and
combining the at least one upmixed portion and the upmixed second portion to acquire a multichannel output signal.

### Claim 29

Non-transitory digital storage medium having a computer program stored thereon to perform the method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, said method comprising:
upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and
downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation,
when said computer program is run by a computer.

### Claim 30

Non-transitory digital storage medium having a computer program stored thereon to perform the method of multichannel decoding, said method comprising:
providing an input downmix representation and parametric data at least for a second portion of the input downmix representation;
the method for generating an output downmix representation from an input downmix representation, wherein at least a portion of the input downmix representation is in accordance with a first downmixing scheme, the method for generating an output downmix representation comprising:
upmixing the input downmix representation of at least the portion of the input downmix representation using an upmixing scheme corresponding to the first downmixing scheme to acquire an at least one upmixed portion; and
downmixing the at least one upmixed portion in accordance with a second downmixing scheme different from the first downmixing scheme to acquire a first downmixed portion representing the output downmix representation for at least the portion of the input downmix representation,
wherein the method comprises the upmixing the input downmix representation for at least the portion of the input downmix representation or only the portion of the input downmix representation in accordance with the upmixing scheme corresponding to the first downmixing scheme to acquire the at least one upmixed portion, and/or upmixing the second portion of the input downmix representation and the parametric data using a second upmixing scheme corresponding to the second downmixing scheme to acquire an upmixed second portion, and
combining the at least one upmixed portion and the upmixed second portion to acquire a multichannel output signal,
when said computer program is run by a computer.

### Claim 31

Apparatus for generating an output downmix representation from an input downmix representation, wherein a first portion of the input downmix representation is in accordance with a first downmixing scheme and a second portion of the input downmix representation is in accordance with the second downmixing scheme, the apparatus comprising:
an upmixer for upmixing the first portion of the input downmix representation using a first upmixing scheme corresponding to the first downmixing scheme to acquire a first upmixed portion and for upmixing the second portion of the input downmix representation using a second upmixing scheme corresponding to the second downmixing scheme to acquire a second upmixed portion; and
a downmixer for downmixing the first upmixed portion and the second upmixed portion in accordance with a third downmixing scheme different from the first downmixing scheme and the second downmixing scheme to acquire the output downmix representation, wherein the output representation for the first portion of the input downmix representation and the output representation for the second portion of the input downmix representation are based on the same downmixing scheme of the input downmix representation.
