# US 2024/0402982 A1 · Artificial reality based system, method and computer program for pre-cueing music audio data

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0402982 A1
- Document ID: US20240402982A1
- Status: Published application
- Kind: A1
- Application number: 18/394,825
- Filed: 2023-12-22
- Published: 2024-12-05
- Earliest claimed: 2023-06-02
- Assignees: Algoriddim GmbH
- Applicants: Algoriddim GmbH
- Inventors: Kariem MORSY; Frederik SEIFFERT; Federico TESSMANN
- Web page: https://patentgrep.com/patent/US20240402982A1

## CPC

- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- G06F 3/014: {Hand-worn input/output arrangements, e.g. data gloves}
- G06F 3/017: {Gesture based interaction, e.g. based on a set of recognized hand gestures (interaction based on gestures traced on a digitiser G06F3/04883)}
- G06T 19/006: {Mixed reality (object pose determination, tracking or camera calibration for mixed reality G06T7/00)}
- G06V 40/28: {Recognition of hand or arm movements, e.g. recognition of deaf sign language (static hand signs G06V40/113)}
- G10H 1/0008: {Associated control or indicating means}
- G10H 1/46: Volume control
- H04R 5/033: Headphones for stereophonic communication {(details thereof, e.g. relating to batteries, cables or control elements H04R1/10)}
- H04S 7/30: {Control circuits for electronic adaptation of the sound field}
- G10H 2210/241: Scratch effects, i.e. emulating playback velocity or pitch manipulation effects normally obtained by a disc-jockey manually rotating a LP record forward and backward
- G10H 2220/201: for movement interpretation, i.e. capturing and recognizing a gesture or a specific kind of movement, e.g. to control a musical instrument
- G10H 2220/321: Garment sensors, i.e. musical control means with trigger surfaces or joint angle sensors, worn as a garment by the player, e.g. bracelet, intelligent clothing
- G10H 2220/401: 3D sensing, i.e. three-dimensional (x, y, z) position or movement sensing
- G10H 2250/035: Crossfade, i.e. time domain amplitude envelope control of the transition between musical sounds or melodies, obtained for musical purposes, e.g. for ADSR tone generation, articulations, medley, remix
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06T: IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- H04S: STEREOPHONIC SYSTEMS

## Abstract

The present invention relates to a system for processing audio data, comprising an audio input unit for receiving audio data representing a piece of music, an audio processing unit for processing the audio data, an audio output unit for playing the audio data, and a gesture detecting apparatus for detecting a pre-cue gesture of a user, wherein the pre-cue gesture comprises moving a hand of the user towards an ear of the user, wherein the audio processing unit is connected to the gesture detecting apparatus such as to change processing of the audio data based on the pre-cue gesture detected by the gesture detecting apparatus.

## Claims

### Claim 1

A system for processing audio data, comprising:
an audio input unit for receiving audio data representing a piece of music;
an audio processing unit for processing the audio data;
an audio output unit for playing the audio data;
a gesture detecting apparatus for detecting a pre-cue gesture of a user; wherein the pre-cue gesture comprises moving a hand of the user towards an ear of the user; and
wherein the audio processing unit is connected to the gesture detecting apparatus such as to change processing of the audio data based on the pre-cue gesture detected by the gesture detecting apparatus.

### Claim 2 (depends on 1)

The system of claim 1, wherein the audio processing unit is configured to increase a volume of the audio data based on the detection of the pre-cue gesture.

### Claim 3 (depends on 1)

The system of claim 1, wherein the gesture detecting apparatus is configured to detect a position of the hand of the user, and wherein the audio processing unit is configured to set a volume of the audio data as being dependent on a hand-ear distance.

### Claim 4 (depends on 1)

The system of claim 1, wherein gesture detecting apparatus classifies a detected hand gesture of the user as being a pre-cue gesture, only if a hand-ear distance is lower than a predetermined first threshold value at least at some point in time during the hand gesture.

### Claim 5 (depends on 1)

The system of claim 1, wherein the system provides a pre-cue mode, wherein, when the pre-cue mode is activated, the audio processing unit is configured to set a volume of the audio data as being dependent on a hand-ear distance, and when the pre-cue mode is deactivated, the audio processing unit is configured to set a volume of the audio data as being independent from the hand-ear distance; and
wherein the system is configured to activate the pre-cue mode when the hand-ear distance is lower than a predetermined second threshold value.

### Claim 6 (depends on 1)

The system of claim 1, wherein the gesture detecting apparatus classifies a detected hand gesture in which a hand of the user is moved towards an ear of the user as not being a pre-cue gesture, if the gesture detecting apparatus detects a substantially simultaneous movement of both hands of the user towards a head of the user.

### Claim 7 (depends on 1)

The system of claim 1, wherein the output unit is a multichannel unit configured to output the audio data as multichannel audio data via at least a left channel and a right channel.

### Claim 8 (depends on 7)

The system of claim 7, wherein the audio processing unit includes a spatial audio unit configured to generate the multichannel audio data as spatial audio data based on a given spatial source position of a sound source;
wherein the gesture detecting apparatus is configured to detect a position in space of the hand of the user; and
wherein the spatial audio unit is configured to generate the multichannel audio data based on the detected position of the hand of the user, such that the audio data are audible for the user as coming from the detected position in space of the hand of the user.

### Claim 9 (depends on 1)

The system of claim 1, comprising:
a first audio input unit for receiving first audio data representing a first piece of music;
a second audio input unit for receiving second audio data representing a second piece of music different from the first piece of music;
wherein the audio processing unit is configured to process at least one of the first audio data and the second audio data; and
wherein the audio processing unit is connected to the gesture detecting apparatus and the audio output unit such as to activate a first pre-cue mode in which the audio output unit outputs exclusively or predominantly the first audio data, when the gesture detecting apparatus detects the pre-cue gesture.

### Claim 10 (depends on 9)

The system of claim 9, wherein the gesture detecting apparatus is configured to detect a first pre-cue gesture comprising moving a hand of the user towards a left ear of the user, and to detect a second pre-cue gesture comprising moving a hand of the user towards a right ear of the user; and
wherein the audio processing unit is configured to activate the first pre-cue mode, when the gesture detecting apparatus detects the first pre-cue gesture, and to activate a second pre-cue mode in which the output unit outputs exclusively or predominantly the second audio data, when the gesture detecting apparatus detects the second pre-cue gesture.

### Claim 11 (depends on 10)

The system of claim 10, wherein the system is configured such that, in the first pre-cue mode, the first audio data are output exclusively or predominantly via the first channel, whereas in the second pre-cue mode, the second audio data are output exclusively or predominantly via the second channel.

### Claim 12 (depends on 9)

The system of claim 9 wherein the audio processing unit is a monitor audio processing unit;
wherein the output unit is a monitor output unit, and wherein the system further comprises:
a main audio processing unit for processing the first audio data and the second audio data, wherein the audio processing unit includes a crossfader unit for setting a first volume level associated to the first audio data and a second volume level associated to the second audio data, and a main mixing unit for mixing the first audio data at the first volume level with the second audio data at the second volume level such as to obtain main output data; and
a main audio output unit for playing the main output data.

### Claim 13 (depends on 12)

The system of claim 12, further comprising:
headphones to be worn by the user or at least one monitor speaker to be directed towards the user, wherein the headphones or the monitor speaker are connected to the monitor output unit; and
at least one PA speaker to be directed towards an audience, wherein the PA speaker is connected to the main audio output unit.

### Claim 14 (depends on 5)

The system of claim 5, wherein the gesture detecting apparatus comprises a touchless user interface, a camera adapted to capture images of the user's hand, and a gesture recognition unit configured to analyze the images captured by the camera to recognize a movement of the hand towards the ear.

### Claim 15 (depends on 14)

The system of claim 14, wherein the camera is arranged such that a field of view of the camera covers an area in front of the user, but does not cover the ear, and wherein the gesture detecting apparatus is configured to detect the pre-cue gesture or to activate the pre-cue mode or to maintain the pre-cue mode active, when detecting the hand of the user leaving the field of view while moving in a direction towards the ear.

### Claim 16 (depends on 1)

The system claim 1, further comprising an artificial reality apparatus adapted to provide visual input for the user to create an artificial reality environment.

### Claim 17 (depends on 1)

The system of claim 1, wherein the gesture detecting apparatus comprises a controller which is a wearable or a handheld controller, wherein the controller comprises at least one sensor element for detecting a hand movement.

### Claim 18 (depends on 3)

The system of claim 3 wherein the gesture detecting apparatus if further configured to increase a volume of the audio data with decreasing hand-ear distance and to continuously decrease the volume of the audio data with increasing hand-ear distance.

### Claim 19

A method comprising the steps of:
providing audio data representing a piece of music;
processing the audio data;
playing the audio data;
detecting a pre-cue gesture of a user;
wherein the pre-cue gesture comprises moving a hand of the user towards an ear of the user; and
wherein processing of the audio data is changed based on a detected pre-cue gesture.

### Claim 20

A non-transitory computer-readable medium embodying program code that, when executed by one or more processors, causes the processors to perform operations comprising:
providing audio data representing a piece of music;
processing the audio data;
playing the audio data;
detecting a pre-cue gesture of a user;
wherein the pre-cue gesture comprises moving a hand of the user towards an ear of the user; and
wherein processing of the audio data is changed based on a detected pre-cue gesture.
