# US 2001/0055398 A1 · Real time audio spatialisation system with high level control

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2001/0055398 A1
- Document ID: US20010055398A1
- Status: Published application
- Kind: A1
- Application number: 09/808,895
- Filed: 2001-03-15
- Published: 2001-12-27
- Earliest claimed: 2000-03-17
- Inventors: Francois Pachet; Olivier Delerue
- Web page: https://patentgrep.com/patent/US20010055398A1

## CPC

- G10K 15/00: Acoustics not otherwise provided for
- H04S 7/30: {Control circuits for electronic adaptation of the sound field}
- H04S 7/303: {Tracking of listener position or orientation}
- H04S 3/002: {Non-adaptive circuits, e.g. manually adjustable or static, for enhancing the sound image or the spatial distribution (control circuits for electronic adaptation of the sound field H04S7/30)}
- H04S 5/00: Pseudo-stereo systems, e.g. in which additional channel signals are derived from monophonic signals by means of phase shifting, time delay or reverberation
- H04S 7/40: {Visual indication of stereophonic sound image}
- H04S 2400/11: Positioning of individual sound objects, e.g. moving airplane, within a sound field (H04S2420/13 takes precedence)
- G10K: SOUND-PRODUCING DEVICES; METHODS OR DEVICES FOR PROTECTING AGAINST, OR FOR DAMPING, NOISE OR OTHER ACOUSTIC WAVES IN GENERAL; ACOUSTICS NOT OTHERWISE PROVIDED FOR
- H04S: STEREOPHONIC SYSTEMS

## Abstract

The invention relates to a system and method for controlling an audio spatialisation in real time, comprising: input means (50) for accessing an audio stream composed of a plurality of audio sources associated to audio tracks, constraint means (3) for receiving and processing constraints expressing rules for a spatialisation of the audio stream, and interface means (2) for entering spatialising commands to the constraint means. The interface means (2) presents at least one user input for effecting a grouped spatialisation command, the command acting on a specified group of audio sources, and the constraint means (3) is programmed to process the group of audio sources as a unitary object for the application of the constraint variables. The group of audio sources typically reflects an internal coherence with respect to the rules for spatialisation. The interface means (2) can be adapted to display: at least one group icon (H) representing a grouped spatialisation command, the icon being positioned according to a topology reflecting a spatialisation and being displaceable by a user, and links between the icons expressing constraints to be applied between the group icons.

## Claims

### Claim 1

System for controlling an audio spatialisation in real time, comprising:
input means (50) for accessing an audio stream composed of a plurality of audio sources associated to audio tracks,
constraint means (3) for receiving and processing constraints expressing rules for a spatialisation of said audio stream, and
interface means (2) for entering spatialising commands to said constraint means,
characterised in that said interface means (2) presents at least one user input for effecting a grouped spatialisation command, said command acting on a specified group of audio sources, and
said constraint means (3) is programmed to process said group of audio sources as a unitary object for the application of said constraint variables.

### Claim 2 (depends on 1)

System according to claim 1, wherein said group of audio sources is identified with a respective group of individually accessible audio tracks.

### Claim 3 (depends on 1, 2)

System according to claim 1 or 2, wherein said group of audio sources reflects an internal coherence with respect to said rules for spatialisation.

### Claim 4 (depends on 1, 2, 3)

System according to any one of claims 1 to 3, wherein said interface means (2) is adapted to display:
at least one group icon (E) representing a grouped spatialisation command, said icon being positioned according to a topology reflecting a spatialisation and being displaceable by a user, and
links between said icons expressing constraints to be applied between said group icons.

### Claim 5 (depends on 1, 2, 3, 4)

System according to any one of claims 1 to 4, further adapted to process global commands through said interface means (2) involving a plurality of groups of audio sources simultaneously.

### Claim 6 (depends on 5)

System according to claim 5, wherein said global commands comprise at least one among:
a balance between a plurality of groups of audio sources (e.g. between two groups respectively corresponding to acoustic and synthetic components), and
a volume level, whereby positions of groups can be changed simultaneously in a proportional manner.

### Claim 7 (depends on 1, 2, 3, 4, 5, 6)

System according to any one of claims 1 to 6, wherein said constraints are one-way constraints, each constraint having a respective set of input and output variables (V) entered by a user through said interface (2).

### Claim 8 (depends on 1, 2, 3, 4, 5, 6, 7)

System according to any one of claims 1 to 7, further adapted to provide a program mode for the recording of mixing constraints entered through said interface means (2) in terms of constraint parameters operative on said groups of audio sources and components of said groups.

### Claim 9 (depends on 8)

System according to claim 8, wherein said interface means (2) is adapted to present each said constraint by a corresponding icon such that they can be linked graphically to an object to be constrained through displayed connections.

### Claim 10 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9)

System according to any one of claims 1 to 9, wherein said constraints are recorded in terms of metadata associated with said audio stream.

### Claim 11 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10)

System according to any one of claims 1 to 10, wherein each constraint is configured as a data string containing a variable part and a constraint part.

### Claim 12 (depends on 11)

System according to claim 11, wherein said variable part expresses at least one among:
a variable type, indicating whether it acts on an audio track or said group,
track identification data,
a variable name,
a variable icon,
individual loudness (for track variables),
initial position data (x,y coordinates).

### Claim 13 (depends on 11, 12)

System according to claim 11 or 12, wherein said constraint part expresses at least one among:
a constraint type,
constrained variables (identification of individual tracks),
a list of input variables,
a list of output variables,
constraint position,
constraint orientations.

### Claim 14 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)

System according to ally one of claims 1 to 13, wherein multiple audio sources for said spatialisation are accessed from a common recorded storage medium (optical disk, hard disk).

### Claim 15 (depends on 14)

System according to claim 14, wherein said constraints are accessed from said common recorded medium as metadata.

### Claim 16 (depends on 15)

System according to claim 15, wherein said metadata and said tracks in which said audio stream is recorded are accessed from a common file, e.g. in accordance with the WAV format.

### Claim 17 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16)

System according to any one of claims 1 to 16, further comprising an audio data and metadata decoder for accessing from a common file audio data and metadata expressing said constraints and recreating therefrom:
a set of audio streams from each individual track contained in said file, and
the specification of said metadata from an encoded format of said file.

### Claim 18 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17)

System according to any one of claims 1 to 17, implemented as an interface to a computer operating system and a sound card.

### Claim 19 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18)

System according to any one of claims 1 to 18, cooperating with a sound card and three-dimensional audio buffering means, said buffering means being physically located in a memory of said sound card so as to benefit from three-dimensional acceleration features of said card.

### Claim 20 (depends on 19)

System according to claim 19, further comprising a waitable timer for controlling writing tasks into said buffering means.

### Claim 21

System according to any one of claims of 1 to 20, wherein said input means is adapted to access audio tracks of said audio stream which are interlaced in a common file.

### Claim 22 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21)

System according to any one of claims 1 to 21, adapted to cooperate with a three-dimensional sound buffer for introducing an orientation constraint.

### Claim 23 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22)

System according to any one of claims 1 to 22, wherein said constraints comprise functional and/or inequality constraints, wherein cyclic constraints are processed through a propagation algorithm by merely checking conflicts.

### Claim 24 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23)

System according to any one of claim 1 to 23, further comprising a means for encoding individual sound sources and a database describing the constraints and relating constraint variables into a common audio file through interlacing.

### Claim 25 (depends on 24)

System according to claim 24, further comprising means for decoding said common audio file in synchronism with said encoding means.

### Claim 26 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25)

System according to any one of claim 1 to 25, further comprising:
a constraint system module for inputting a database describing the constraints and relating constraint variables for each music title, thereby creating spatialisation commands; and
a spatialisation controller module for inputting said set of audio streams given by encoding means, and spatialisation commands given by said constraint system module.

### Claim 27 (depends on 26)

System according to claim 26, further comprising three-dimensional sound buffer means, in which a writing task and a reading task for each sound source are synchronised, said means thereby relaying said audio stream coming from an audio file into a spatialisation controller module and relaying said database describing the constraints and relating constraint variables for each music title into said constraint module means.

### Claim 28 (depends on 26, 27)

System according to claim 26 or 27, wherein said spatialisation controller module further comprises a scheduler means for connecting said constraint system module and said spatialisation controller module.

### Claim 29 (depends on 27, 28)

System according to any one of claims 27 to 28, wherein said spatialisation controller module comprises static audio secondary buffer means.

### Claim 30 (depends on 27, 28, 29)

System according to any one of claims 27 to 29, further comprising a timer means for waking up said writing task at predetermined intervals.

### Claim 31 (depends on 26, 27, 28, 29, 30)

System according to any one of claims 26 to 30, wherein said spatialisation controller module is a remote controllable mixing device.

### Claim 32 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31)

System according to any one of claims 1 to 31, wherein said constraint means (3) is configured to execute a test algorithm.

### Claim 33 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32)

A spatialisation apparatus comprising:
a personal computer having a data reader for reading from a common data medium both audio stream data and data representative of constraints for spatialisation, and
an audio spatialisation system according to any one of claims 1 to 32 having its input means adapted to receive data from said data reader.

### Claim 34 (depends on 33)

Spatialisation apparatus according to claim 33, wherein said computer comprises a three-dimensional sound buffer for storing contents extracted from data reader.

### Claim 35 (depends on 34)

Spatialisation apparatus according to claim 34, wherein said sound buffer is controlled through a dynamic link library (DLL).

### Claim 36 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32)

A storage medium containing data specifically adapted for exploitation by an audio spatialisation control system according to any one of claims 1 to 32, comprising a plurality of tracks forming an audio stream and data representative of said processing constraints.

### Claim 37 (depends on 36)

Storage medium according to claim 36, wherein said data representative of said processing constraints and said plurality of tracks are recorded in a common file.

### Claim 38 (depends on 36, 37)

Storage medium according claim 36 or 37, wherein said data representative of said processing constraints are recorded as metadata with respect to said tracks.

### Claim 39 (depends on 36, 37, 38)

Storage medium according to any one of claims 36 to 38, wherein said tracks are interlaced.

### Claim 40 (depends on 35, 36, 37, 38, 39)

Storage medium according to any one of claims 35 to 39 in the form of any digital storage medium, such as a CD-ROM, DVD ROM or minidisk.

### Claim 41 (depends on 36, 37, 38, 39, 40)

Storage medium according to any one of claims 36 to 40 in the form of a computer hard disk.

### Claim 42 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32)

A computer program product loadable into the internal memory unit of a general-purpose computer, comprising a software code unit for coding the system according to any one of claims 1 to 32 and implementing the means described in said system, when said computer program product is run on a computer.

### Claim 43

A method of controlling an audio spatialisation, comprising the steps of:
accessing an audio stream composed of a plurality of audio sources associated to audio tracks,
receiving and processing constraints expressing rules for a spatialisation of said audio stream, and
entering spatialising commands to said constraint means through an interface,
characterised in that at least one user input is provided for effecting a grouped spatialisation command, said command acting on a specified group of audio sources, and
said group of audio sources is processed as a unitary object for the application of said constraint variables.
