# US 11,122,381 B2 · Spatial audio signal processing

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,122,381 B2
- Document ID: US11122381B2
- Status: Granted
- Kind: B2
- Application number: 16/649,433
- Filed: 2018-10-01
- Published: 2020-09-10
- Granted: 2021-09-14
- Earliest claimed: 2017-10-09
- Assignees: Nokia Technologies Oy
- Applicants: Nokia Technologies Oy
- Inventors: Antti Eronen; Arto Lehtiniemi; Jussi Leppanen; Tapani Pihlajakuja
- Web page: https://patentgrep.com/patent/US11122381B2

## CPC

- G10L 25/48: specially adapted for particular use
- H04S 7/303: {Tracking of listener position or orientation}
- G01S 3/80: using ultrasonic, sonic or infrasonic waves
- H04S 7/302: {Electronic adaptation of stereophonic sound system to listener position or orientation (H04S7/301 takes precedence)}
- G10L 25/03: characterised by the type of extracted parameters
- H04R 2201/40: Details of arrangements for obtaining desired directional characteristic by combining a number of identical transducers covered by H04R1/40 but not provided for in any of its subgroups
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/01: Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04S: STEREOPHONIC SYSTEMS
- G01S: RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)

## Abstract

An apparatus for spatial audio signal processing, the apparatus including at least one processor configured to: receive captured audio content, the captured audio content captured within a capture environment with defined geometry; determine audio directions and distances associated with the captured audio content; determine a listening space area geometry; map the audio directions and distances associated with the captured audio content relative to the defined geometry to the listening space area geometry; and synthesize a volumetric audio based on the mapped audio directions and distances to generate an audio content experience within the listening space area geometry.

## Claims

### Claim 1

An apparatus for spatial audio signal processing, the apparatus comprising:
at least one processor; and
at least one non-transitory memory and computer program code, wherein the at least one memory and the computer program code are configured to, with the at least one processor, cause the apparatus to:
receive captured audio content, wherein the captured audio content is captured within a capture environment with defined geometry, wherein the captured audio content comprises at least audio associated with a first audio source;
determine a first audio direction and distance, associated with the first audio source of the captured audio content, relative to the defined geometry of the capture environment;
determine a listening space area geometry;
map the first audio direction and distance to a distance and a direction relative to the listening space area geometry, wherein at least one of the distance and/or direction relative to the determined listening space geometry is different than the first distance and/or the first direction associated with the first audio source; and
synthesize a volumetric audio based on at least the mapped audio direction and distance to generate an audio content experience within the listening space area geometry.

### Claim 2 (depends on 1)

The apparatus as claimed in claim 1, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive the captured audio content are further configured to cause the apparatus to receive at least one of:
professional audio content, the professional audio content captured within the capture environment with the defined geometry; or
personal audio content, the personal audio content captured within the capture environment with the defined geometry.

### Claim 3 (depends on 1)

The apparatus as claimed in claim 1, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to determine at least the first audio direction and distance associated with the captured audio content are further configured to cause the apparatus to one of:
receive at least the first audio direction and distance associated with the captured audio content; or
analyse the captured audio content to generate at least the first audio direction and distance associated with the captured audio content.

### Claim 4 (depends on 1)

The apparatus as claimed in claim 1, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive the captured audio content are further configured to cause the apparatus to receive:
professional audio content, the professional audio content captured within the capture environment with the defined geometry; and
personal audio content, the personal audio content captured within the capture environment with the defined geometry, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to determine at least the first audio direction and distance associated with the captured audio content are further configured to cause the apparatus to:
generate a local personal audio content by removing a common part between the captured professional audio content and the captured personal audio content from the captured personal audio content; and
analyse the local personal audio content to determine audio directions and distances associated with the local personal audio content.

### Claim 5 (depends on 4)

The apparatus as claimed in claim 4, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to determine at least the first audio direction and distance associated with the captured audio content are further configured to cause the apparatus to:
generate a far professional audio content by removing the common part between the captured professional audio content and the captured personal audio content from the captured professional audio content; and
analyse the far professional audio content to determine audio directions and distances associated with the far professional audio content.

### Claim 6 (depends on 5)

The apparatus as claimed in claim 5, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to synthesize the volumetric audio based on at least the mapped audio direction and distance to generate the audio content experience within the listening space area geometry are further configured to cause the apparatus to:
synthesize a personal volumetric audio using the local personal audio content based on at least the mapped audio direction and distance to generate a local personal audio content experience within the listening space area geometry;
synthesize a professional volumetric audio using the far professional audio content based on at least the mapped audio direction and distance to generate a far professional audio content experience within the listening space area geometry associated with the professional audio content; and
mix the professional volumetric audio and the personal volumetric audio to generate the volumetric audio to generate the audio content experience.

### Claim 7 (depends on 6)

The apparatus as claimed in claim 6, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to synthesize the professional volumetric audio are further configured to cause the apparatus to synthesize the professional volumetric audio based on a determined listening position.

### Claim 8 (depends on 6)

The apparatus as claimed in claim 6, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to synthesize the personal volumetric audio are further configured to cause the apparatus to:
determine a listening position; and
synthesize the personal volumetric audio based on the listening position.

### Claim 9 (depends on 4)

The apparatus as claimed in claim 4, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to receive captured personal audio content are further configured to cause the apparatus to receive at least two separate captured personal audio contents, and wherein the at least one memory and the computer program code are further configured to, with the at least one processor, cause the apparatus to selectively switch between the at least two separate captured personal audio contents based on a user interface input.

### Claim 10 (depends on 1)

The apparatus as claimed in claim 1, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to determine the listening space area geometry are further configured to cause the apparatus to:
determine within the listening space area geometry a listening space area geometry associated with the captured audio content.

### Claim 11 (depends on 1)

The apparatus as claimed in claim 1, wherein the at least one memory and the computer program code configured to, with the at least one processor, cause the apparatus to determine the listening space area geometry are further configured to cause the apparatus to:
receive information from at least one sensor located within the listening space; and
determine the listening space area geometry from the information from the at least one sensor located within the listening space.

### Claim 12 (depends on 11)

The apparatus as claimed in claim 11, further comprising the at least one sensor, wherein the at least one sensor comprises at least one of:
a camera pair;
a phase or time difference based laser range detector; or
a light field camera.

### Claim 13 (depends on 2)

The apparatus as claimed in claim 2, wherein the captured professional audio content comprises at least one of:
an audio signal captured by a close microphone within the capture environment;
an audio signal captured by a public address microphone within the capture environment;
an audio signal captured from a mixing desk within the capture environment; or
an audio signal captured by a spatial microphone array within the capture environment.

### Claim 14

A method for spatial audio signal processing comprising:
receiving captured audio content, the captured audio content captured within a capture environment with defined geometry, wherein the captured audio content comprises at least audio associated with a first audio source;
determining a first audio direction and distance, associated with the first audio source of the captured audio content, relative to the defined geometry of the capture environment;
determining a listening space area geometry;
mapping the first audio direction and distance to distance to a distance and a direction relative to the listening space area geometry, wherein at least one of the distance and/or direction relative to the determined listening space geometry is different than the first distance and/or the first direction associated with the first audio source; and
synthesizing a volumetric audio based on at least the mapped audio direction and distance to generate an audio content experience within the listening space area geometry.

### Claim 15 (depends on 14)

The method as claimed in claim 14, wherein receiving the captured audio content further comprises at least one of:
receiving professional audio content, the professional audio content captured within the capture environment with the defined geometry; or
receiving personal audio content, the personal audio content captured within the capture environment with the defined geometry.

### Claim 16 (depends on 14)

The method as claimed in claim 14, wherein determining at least the first audio direction and distance associated with the captured audio content comprises one of:
receiving at least the first audio direction and distance associated with the captured audio content; or
analysing the captured audio content to generate at least the first audio direction and distance associated with the captured audio content.

### Claim 17 (depends on 14)

The method as claimed in claim 14, wherein receiving the captured audio content comprises:
receiving professional audio content, the professional audio content captured within the capture environment with the defined geometry; and
receiving personal audio content, the personal audio content captured within the capture environment with the defined geometry, wherein determining at least the first audio direction and distance associated with the captured audio content comprises:
generating a local personal audio content by removing a common part between the captured professional audio content and the captured personal audio content from the captured personal audio content; and
analysing the local personal audio content to determine audio directions and distances associated with the local personal audio content.

### Claim 18 (depends on 17)

The method as claimed in claim 17, wherein determining at least the first audio direction and distance associated with the captured audio content comprises:
generating a far professional audio content by removing the common part between the captured professional audio content and the captured personal audio content from the captured professional audio content; and
analysing the far professional audio content to determine audio directions and distances associated with the far professional audio content.

### Claim 19 (depends on 18)

The method as claimed in claim 18, wherein synthesizing the volumetric audio based on at least the mapped audio direction and distance to generate the audio content experience within the listening space area geometry comprises:
synthesizing a personal volumetric audio using the local personal audio content based on the mapped audio direction and distance to generate a local personal audio content experience within the listening space area geometry;
synthesizing a professional volumetric audio using the far professional audio content based on the mapped audio direction and distance to generate a far professional audio content experience within the listening space area geometry associated with the professional audio content; and
mixing the professional volumetric audio and the personal volumetric audio to generate the volumetric audio to generate the audio content experience.

### Claim 20 (depends on 19)

The method as claimed in claim 19, further comprising at least one of:
determining a listening position; and
synthesizing the personal volumetric audio based on the listening position.