# US 2021/0343300 A1 · Apparatus and Method for Encoding a Spatial Audio Representation or Apparatus and Method for Decoding an Encoded Audio Signal Using Transport Metadata and Related Computer Programs

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0343300 A1
- Document ID: US20210343300A1
- Status: Published application
- Kind: A1
- Application number: 17/375,465
- Filed: 2021-07-14
- Published: 2021-11-04
- Earliest claimed: 2019-01-21
- Assignees: Fraunhofer-Gesellschaft zur Forderung der Angewandlen Forschung EV
- Applicants: Fraunhofer-Gesellschaft zur Förderung der angewandlen Forschung e.V.
- Inventors: Fabian KÜCH; Oliver THIERGART; Guillaume FUCHS; Stefan DÖHLA; Alexandre BOUTHÉON; Jürgen HERRE; Stefan BAYER
- Web page: https://patentgrep.com/patent/US20210343300A1

## CPC

- G10L 19/008: Multichannel audio signal coding or decoding using interchannel correlation to reduce redundancy, e.g. joint-stereo, intensity-coding or matrixing
- G10L 19/032: Quantisation or dequantisation of spectral components
- G10L 19/26: Pre-filtering or post-filtering
- H04S 1/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- H04S 7/307: {Frequency adjustment, e.g. tone control (H04S7/301 takes precedence)}
- H04S 2420/11: Application of ambisonics in stereophonic audio systems
- H04S 2420/13: Application of wave-field synthesis in stereophonic audio systems
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04S: STEREOPHONIC SYSTEMS

## Abstract

An apparatus for encoding a spatial audio representation representing an audio scene to obtain an encoded audio signal includes: a transport representation generator for generating a transport representation from the spatial audio representation, and for generating transport metadata related to the generation of the transport representation or indicating one or more directional properties of the transport representation; and an output interface for generating the encoded audio signal, the encoded audio signal including information on the transport representation, and information on the transport metadata.

## Claims

### Claim 1

An apparatus for encoding a spatial audio representation representing an audio scene to acquire an encoded audio signal, the apparatus comprising:
a transport representation generator for generating a transport representation from the spatial audio representation, and for generating transport metadata related to the generation of the transport representation or indicating one or more directional properties of the transport representation; and
an output interface for generating the encoded audio signal, the encoded audio signal comprising information on the transport representation, and information on the transport metadata.

### Claim 2 (depends on 1)

The apparatus of claim 1, further comprising a parameter processor for deriving spatial parameters from the spatial audio representation,
wherein the output interface is configured for generating the encoded audio signal such that the encoded audio signal additionally comprises information on the spatial parameters.

### Claim 3 (depends on 1)

The apparatus of claim 1,
wherein the spatial audio representation is a first order Ambisonics or higher order Ambisonics representation comprising a multitude of coefficient signals, or a multi-channel representation comprising a plurality of audio channels,
wherein the transport representation generator is configured to select one or more coefficient signals from the first order Ambisonics or higher order Ambisonics representation or to combine coefficients from the higher order Ambisonics or first order Ambisonics representation, or wherein the transport representation generator is configured to select one or more audio channels from the multichannel representation or to combine two or more audio channels from the multichannel representation, and
wherein the transport representation generator is configured to generate, as the transport metadata, information indicating which specific one or more coefficient signals or audio channels have been selected, or information how the two or more coefficients signals or audio channels have been combined, or which ones of the first order Ambisonics or higher order Ambisonics coefficient signals or audio channels have been combined.

### Claim 4 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to determine, whether a majority of sound energy is located in a horizontal plane, or
wherein only an omnidirectional coefficient signal, an X coefficient signal and a Y coefficient signal are selected as the transport representation in response to the determination or in response to an audio encoder setting, and
wherein the transport representation generator is configured to determine the transport metadata so that the transport metadata comprises an information on the selection of the coefficient signals.

### Claim 5 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to determine, whether a majority of sound energy is located in a x-z plane, or
wherein only an omnidirectional coefficient signal, a X coefficient signal and a Z coefficient signal are selected as the transport representation in response to the determination or in response to an audio encoder setting, and
wherein the transport representation generator is configured to determine the transport metadata so that the transport metadata comprises an information on the selection of the coefficient signal.

### Claim 6 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to determine, whether a majority of sound energy is located in a y-z plane, or
wherein only an omnidirectional coefficient signal, a Y coefficient signal and a Z coefficient signal are selected as the transport representation in response to the determination or in response to an audio encoder setting, and
wherein the transport representation generator is configured to determine the transport metadata so that the transport metadata comprises an information on the selection of the coefficient signals.

### Claim 7 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to determine whether a dominant sound energy originates from a specific sector or hemisphere such as a left or right hemisphere or a forward or backward hemisphere, or
wherein the transport representation generator is configured to generate a first transport signal from the specific sector or hemisphere, where a dominant sound energy originates or in response to an audio encoder setting, and a second transport signal from a different sector or hemisphere such as the sector or hemisphere comprising an opposite direction with respect to a reference location and with respect to the specific sector or hemisphere, and
wherein the transport representation generator is configured to determine the transport metadata so that the transport metadata comprises information identifying the specific sector or hemisphere, or identifying the different sector or hemisphere.

### Claim 8 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to combine coefficient signals of the spatial audio representation so that a first resulting signal being a first transport signal corresponds to a directional microphone signal directed to a specific sector or hemisphere, and a second resulting signal being a second transport signal corresponds to a directional microphone signal directed to a different sector or hemisphere.

### Claim 9 (depends on 1)

The apparatus of claim 1, further comprising a user interface for receiving a user input,
wherein the transport representation generator is configured to generate the transport representation based on the user input received at the user interface, and
wherein the transport representation generator is configured to generate the transport metadata so that the transport metadata comprises information on the user input.

### Claim 10 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to generate the transport representation and the transport metadata in a time-variant or frequency-dependent way, so that the transport representation and the transport metadata for a first frame is different from the transport representation and the transport metadata for a second frame, or so that the transport representation and the transport metadata for a first frequency band is different from a transport representation and the transport metadata for a second different frequency band.

### Claim 11 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to generate one or two transport signals by a weighted combination of two or more than two coefficient signals of the spatial audio representation, and
wherein the transport representation generator is configured to calculate the transport metadata so that the transport metadata comprises information on weights used in the weighted combination, or information on an azimuth and/or elevation angle as a look direction of a generated directional microphone signal, or information on a shape parameter indicating a directional characteristic of a directional microphone signal.

### Claim 12 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to generate quantitative transport metadata, to quantize the quantitative transport metadata to acquire quantized transport metadata, and to entropy encode the quantized transport metadata, and wherein the output interface is configured to comprise the encoded transport metadata into the encoded audio signal.

### Claim 13 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to transform the transport metadata into a table index or a preset parameter, and
wherein the output interface is configured to comprise the table index or the preset parameter into the encoded audio signal.

### Claim 14 (depends on 1)

The apparatus of claim 1,
wherein the spatial audio representation comprises at least two audio signals and spatial parameters,
wherein a parameter processor is configured to derive the spatial parameters from the spatial audio representation by extracting the spatial parameters from the spatial audio representation,
wherein the output interface is configured to comprise information on the spatial parameters into the encoded audio signal or to comprise information on processed spatial parameters derived from the spatial parameters into the encoded audio signal, or
wherein the transport representation generator is configured to select a subset of the at least two audio signals as the transport representation and to generate the transport metadata so that the transport metadata indicates the selection of the subset, or to combine the at least two audio signals or a subset of the at least two audio signals and to calculate the transport metadata such that the transport metadata comprises information on the combination of the audio signals performed for calculating the transport representation of the spatial audio representation.

### Claim 15 (depends on 1)

The apparatus of claim 1,
wherein the spatial audio representation comprises a set of at least two microphone signals acquired by a microphone array,
wherein the transport representation generator is configured to select one or more specific microphone signals associated with specific locations or with specific microphones of the microphone array, and
wherein the transport metadata comprises information on the specific locations or the specific microphones or on a microphone distance between locations associated with selected microphone signals, or information on a microphone orientation of a microphone associated with a selected microphone signal, or information on microphone directional patterns of microphone signals associated with selected microphones.

### Claim 16 (depends on 15)

The apparatus of claim 15,
wherein the transport representation generator is configured
to select one or more signals of the spatial audio representation in accordance with a user input received by a user interface,
to perform an analysis of the spatial audio representation with respect to which location comprises which sound energy and to select one or more signals of the spatial audio representation in accordance with an analysis result, or
to perform a sound source localization and to select one or more signals of the spatial audio representation in accordance with a result of the sound source localization.

### Claim 17 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to select all signals of a spatial audio representation, and
wherein the transport representation generator is configured to generate the transport metadata so that the transport metadata identifies a microphone array, from which the spatial audio representation is derived.

### Claim 18 (depends on 1)

The apparatus of claim 1,
wherein the transport representation generator is configured to combine audio signals comprised in the spatial audio representation using spatial filtering or beamforming, and
wherein the transport representation generator is configured to comprise information on the look direction of the transport representation or information on beamforming weights used in calculating the transport representation into the transport metadata.

### Claim 19 (depends on 1)

The apparatus of claim 1,
wherein the spatial audio representation is a description of a sound field related to a reference position, and
wherein a parameter processor is configured to derive spatial parameters from the spatial audio representation, wherein the spatial parameters define time-variant or frequency-dependent parameters on a direction of arrival of sound at the reference position or time-variant or frequency-dependent parameters on a diffuseness of the sound field at the reference position, or
wherein the transport representation generator comprises a downmixer for generating, as the transport representation, a downmix representation comprising a second number of individual signals being smaller than a first number of individual signals comprised in the spatial audio representation, wherein the downmixer is configured to select a subset of the individual signals comprised in the spatial audio representation or to combine the individual signals comprised in the spatial audio representation in order to decrease the first number of signals to the second number of signals.

### Claim 20 (depends on 1)

The apparatus of claim 1,
wherein a parameter processor comprises a spatial audio analyzer for deriving the spatial parameters from the spatial audio representation by performing an audio signal analysis, and
wherein the transport representation generator is configured to generate the transport representation based on the result of the spatial audio analyzer, or wherein the transport representation comprises a core encoder for core encoding one or more audio signals of the transport signals of the transport representation, or
wherein the parameter processor is configured to quantize and entropy encode the spatial parameters, and
wherein the output interface is configured to comprise a core-encoded transport representation as the information on the transport representation into the encoded audio signal or to comprise the entropy encoded spatial parameters as the information on spatial parameters into the encoded audio signal.

### Claim 21

An apparatus for decoding an encoded audio signal, comprising:
an input interface for receiving the encoded audio signal comprising information on a transport representation and information on transport metadata; and
a spatial audio synthesizer for synthesizing a spatial audio representation using the information on the transport representation and the information on the transport metadata.

### Claim 22 (depends on 21)

The apparatus of claim 21, wherein the input interface is configured for receiving the encoded audio signal additionally comprising information on spatial parameters, and
wherein the spatial audio synthesizer is configured for synthesizing the spatial audio representation additionally using the information on the spatial parameters.

### Claim 23 (depends on 21)

The apparatus of claim 21, wherein the spatial audio synthesizer comprises:
a core decoder for core decoding two or more encoded transport signals representing the information on the transport representation to acquire two or more decoded transport signals, or
wherein the spatial audio synthesizer is configured to calculate a first order Ambisonics or a higher order Ambisonics representation or a multi-channel signal or an object representation or a binaural representation of the spatial audio representation, or
wherein the spatial audio synthesizer comprises a metadata decoder for decoding the information on the transport metadata to derive the decoded transport metadata or for decoding information on spatial parameters to acquire decoded spatial parameters.

### Claim 24 (depends on 21)

The apparatus of claim 21,
wherein the spatial audio representation comprises a plurality of component signals,
wherein the spatial audio synthesizer is configured to determine, for a component signal of the spatial audio representation, a reference signal using the information on the transport representation and the information on the transport metadata, and
to calculate the component signal of the spatial audio representation using the reference signal and information on spatial parameters, or to calculate the component signal of the spatial audio representation using the reference signal.

### Claim 25 (depends on 22)

The apparatus of claim 22,
wherein the spatial parameters comprise at least one of the time-variant or frequency-dependent direction of arrival or diffuseness parameters,
wherein the spatial audio synthesizer is configured to perform a directional audio coding synthesis using the spatial parameters to generate the plurality of different components of the spatial audio representation,
wherein the first component of the spatial audio representation is determined using one of the at least two transport signals or a first combination of the at least two transport signals,
wherein a second component of the spatial audio representation is determined using another one of the at least two transport signals or a second combination of the at least two transport signals,
wherein the spatial audio synthesizer is configured to perform a determination of the one or the different one of the at least two transport signals or to perform a determination of the first combination or the different second combination in accordance with the transport metadata.

### Claim 26 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata indicates a first transport signal as referring to a first sector or hemisphere related to a reference position of the spatial audio representation and a second transport signal as referring to a second different sector or hemisphere related to the reference position of the spatial audio representation,
wherein the spatial audio synthesizer is configured to generate a component signal of the spatial audio representation associated with the first sector or hemisphere using the first transport signal and without using the second transport signal, or wherein the spatial audio synthesizer is configured to generate another component signal of the spatial audio representation associated with the second sector or hemisphere using the second transport signal and not using the first transport signal, or
wherein the spatial audio synthesizer is configured to generate a component signal associated with the first sector or hemisphere using a first combination of the first and the second transport signal, or to generate a component signal associated with a different second sector or hemisphere using a second combination of the first and the second transport signals, wherein the first combination is influenced by the first transport signal stronger than the second combination, or wherein the second combination is influenced by the second transport signal stronger than the first combination.

### Claim 27 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata comprises information on a directional characteristic associated with transport signals of the transport representation,
wherein the spatial audio synthesizer is configured to calculate virtual microphone signals using first order Ambisonics or higher order Ambisonics signals, loudspeaker positions and the transport metadata, or
wherein the spatial audio synthesizer is configured to determine the directional characteristic of the transport signals using the transport metadata and to determine a first order Ambisonics or a higher order Ambisonics component from the transport signals in line with the determined directional characteristics of the transport signals, or
to determine a first order Ambisonics or higher order Ambisonics component not associated with the directional characteristics of the transport signals in accordance with a fallback process.

### Claim 28 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata comprises an information on the first look direction associated with a first transport signal, and an information on a second look direction associated with a second transport signal,
wherein the spatial audio synthesizer is configured to select a reference signal for the calculation of a component signal of the spatial audio representation based on the transport metadata and the position of a loudspeaker associated with the component signal of the spatial audio representation.

### Claim 29 (depends on 28)

The apparatus of claim 28,
wherein the first look direction indicates a left or a front hemisphere, wherein the second look direction indicates a right or a back hemisphere,
wherein, for the calculation of a component signal for a loudspeaker in the left hemisphere, the first transport signal and not the second transport signal is used, or wherein for the calculation of a loudspeaker signal in the right hemisphere, the second transport signal and not the first transport signal is used, or
wherein for the calculation of a loudspeaker in a front hemisphere, the first transport signal and not the second transport signal is used, or wherein for the calculation of a loudspeaker in a back hemisphere, the second transport signal and not the first transport signal is used, or
wherein for the calculation of a loudspeaker in a center region, a combination of the left transport signal and the second transport signal is used, or wherein for the calculation of a loudspeaker signal associated with a loudspeaker in a region between the front hemisphere and the back hemisphere, a combination of the first transport signal and the second transport signal is used.

### Claim 30 (depends on 21)

The apparatus of claim 21,
wherein the information on the transport metadata indicates, as a first look direction, a left direction for a left transport signal and indicates, as a second look direction, a right look direction for a second transport signal,
wherein the spatial audio synthesizer is configured to calculate a first Ambisonics component by adding the first transport signal and the second transport signal, or to calculate a second Ambisonics component by subtracting the first transport signal and the second transport signal, or wherein another Ambisonics component is calculated using a sum of the first transport signal and the second transport signal.

### Claim 31 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata indicates, for a first transport signal, a front look direction and indicates, for a second transport signal, a back look direction,
wherein the spatial audio synthesizer is configured to calculate a first order Ambisonics component for an x direction by performing the calculation of a difference between the first and the second transport signals, and to calculate an omnidirectional first order Ambisonics component using an addition of the first transport signal and the second transport signal, and
to calculate another first order Ambisonics component using a sum of the first transport signal and the second transport signal.

### Claim 32 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata indicate information on weighting coefficients or look directions of transport signals of the transport representation,
wherein the spatial audio synthesizer is configured to calculate different first order Ambisonics components of the spatial audio representation using the information on the look direction or the weighting coefficients, using the transport signals and the spatial parameters, or wherein the spatial audio synthesizer is configured to calculate different first order Ambisonics components of the spatial audio representation using the information on the look direction or the weighting coefficients, and using the transport signals.

### Claim 33 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata comprise information on the transport signals being derived from microphone signals at two different positions or with different look directions,
wherein the spatial audio synthesizer is configured to select a reference signal that comprises a position that is closest to a loudspeaker position, or to select a reference signal that comprises a closest look direction with respect to the direction from a reference position of the spatial audio representation and a loudspeaker position, or
wherein the spatial audio synthesizer is configured to perform a linear combination with the transport signals to determine a reference signal for a loudspeaker being placed between two look directions indicated by the transport metadata.

### Claim 34 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata comprises information on a distance between microphone positions associated with the transport signals,
wherein the spatial audio synthesizer comprises a diffuse signal generator, and wherein the diffuse signal generator is configured to control an amount of a decorrelated signal in a diffuse signal generated by the diffuse signal generator using the information on the distance, so that, for a first distance, a higher amount of decorrelated signal is comprised in the diffuse signal compared to an amount of decorrelated signal for a second distance, wherein the first distance is lower than the second distance, or
wherein the spatial audio synthesizer is configured to calculate, for a first distance between the microphone positions, a component signal for the spatial audio representation using an output signal of a decorrelation filter configured for decorrelating a reference signal or a scaled reference signal and the reference signal weighted using a gain derived from a sound direction of arrival information and to calculate, for a second distance between the microphone positions, a component signal for the spatial audio representation using the reference signal weighted using a gain derived from a sound direction of arrival information without any decorrelation processing, the second distance being greater than the first distance or being greater than a distance threshold.

### Claim 35 (depends on 21)

The apparatus of claim 21,
wherein the transport metadata comprises information on a beamforming or a spatial filtering associated with the transport signals of the transport representation, and
wherein the spatial audio synthesizer is configured to generate a loudspeaker signal for a loudspeaker using the transport signal comprising a look direction being closest to a look direction from a reference position of the spatial audio representation to the loudspeaker.

### Claim 36 (depends on 21)

The apparatus of claim 21,
wherein the spatial audio synthesizer is configured to determine component signals of the spatial audio representation as a combination of a direct sound component and a diffuse sound component, wherein the direct sound component is acquired by scaling a reference signal with a factor depending on a diffuseness parameter or a directional parameter, wherein the directional parameter depends on a direction of arrival of sound, wherein the determination of the reference signal is performed based on the information on the transport metadata, and wherein the diffuse sound component is determined using the same reference signal and the diffuseness parameter.

### Claim 37 (depends on 21)

The apparatus of claim 21,
wherein the spatial audio synthesizer is configured to determine component signals of the spatial audio representation as a combination of a direct sound component and a diffuse sound component, wherein the direct sound component is acquired by scaling a reference signal with a factor depending on a diffuseness parameter or a directional parameter, wherein the directional parameter depends on a direction of arrival of sound, wherein the determination of the reference signal is performed based on the information on the transport metadata, and wherein the diffuse sound component is determined using a decorrelation filter, the same reference signal, and the diffuseness parameter.

### Claim 38 (depends on 21)

The apparatus of claim 21, wherein the transport representation comprises at least two different microphone signals,
wherein the transport metadata comprises information indicating, whether the at least two different microphone signals are at least one of omnidirectional signals, dipole signals or cardioid signals, and
wherein the spatial audio synthesizer is configured for adapting a reference signal determination to the transport metadata to determine, for components of the spatial audio representation, individual reference signals and for calculating the respective component using the individual reference signal determined for the respective component.

### Claim 39

A method for encoding a spatial audio representation representing an audio scene to acquire an encoded audio signal, the method comprising:
generating a transport representation from the spatial audio representation;
generating transport metadata related to the generation of the transport representation or indicating one or more directional properties of the transport representation; and
generating the encoded audio signal, the encoded audio signal comprising information on the transport representation, and information on the transport metadata.

### Claim 40 (depends on 39)

The method of claim 39, further comprising deriving spatial parameters from the spatial audio representation, and wherein the encoded audio signal additionally comprises information on the spatial parameters.

### Claim 41

The method for decoding an encoded audio signal, the method comprising:
receiving the encoded audio signal comprising information on a transport representation and information on transport metadata; and
synthesizing a spatial audio representation using the information on the transport representation and the information on the transport metadata.

### Claim 42 (depends on 41)

The method of claim 41, further comprising receiving information on spatial parameters, and wherein the synthesizing additionally uses the information on the spatial parameters.

### Claim 43

A non-transitory digital storage medium having a computer program stored thereon to perform the method for encoding a spatial audio representation representing an audio scene to acquire an encoded audio signal, the method comprising:
generating a transport representation from the spatial audio representation;
generating transport metadata related to the generation of the transport representation or indicating one or more directional properties of the transport representation; and
generating the encoded audio signal, the encoded audio signal comprising information on the transport representation, and information on the transport metadata,
when said computer program is run by a computer.

### Claim 44

A non-transitory digital storage medium having a computer program stored thereon to perform the method for decoding an encoded audio signal, the method comprising:
receiving the encoded audio signal comprising information on a transport representation and information on transport metadata; and
synthesizing a spatial audio representation using the information on the transport representation and the information on the transport metadata,
when said computer program is run by a computer.
