# US 2021/0279030 A1 · Method and device for processing, playing and/or visualizing audio data, preferably based on ai, in particular decomposing and recombining of audio data in real-time

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0279030 A1
- Document ID: US20210279030A1
- Status: Published application
- Kind: A1
- Application number: 16/892,063
- Filed: 2020-06-03
- Published: 2021-09-09
- Earliest claimed: 2020-03-06
- Assignees: Algoriddim GmbH
- Applicants: Algoriddim GmbH
- Inventors: Kariem MORSY; Federico TESSMANN; Christoph TESCHNER
- Web page: https://patentgrep.com/patent/US20210279030A1

## CPC

- G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path}
- H04R 29/008: {Visual indication of individual signal levels (visual indication of stereophonic sound image H04S7/40)}
- G06F 3/0482: Interaction with lists of selectable items, e.g. menus
- G06F 3/04847: Interaction techniques to control parameter settings, e.g. interaction with sliders or dials
- G06F 3/04883: for inputting data by handwriting, e.g. gesture or text
- G06F 3/04886: by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10H 1/0008: {Associated control or indicating means}
- G10H 1/0091: {Means for obtaining special acoustic effects (combined with modulation G10H1/043)}
- G10H 1/08: by combining tones (G10H1/14, G10H1/16 take precedence; chord G10H1/38)
- G10H 1/40: Rhythm
- G10H 1/46: Volume control
- G10L 21/028: using properties of sound source
- G10L 21/0316: by changing the amplitude
- G10L 21/034: Automatic adjustment
- G10L 21/043: by changing speed
- G10L 25/30: using neural networks
- G10L 25/51: for comparison or discrimination
- G11B 20/10527: {Audio or video recording; Data buffering arrangements (G11B20/12 - G11B20/18 take precedence)}
- G11B 27/105: {of operating discs}
- H04H 60/05: Mobile studios
- H04N 21/439: Processing of audio elementary streams
- H04R 3/00: Circuits for transducers (arrangements for producing a reverberation or echo sound G10K15/08; amplifiers H03F)
- H04R 3/12: for distributing signals to two or more loudspeakers {(specially adapted for hearing aids H04R25/407)}
- H04S 1/007: {in which the audio signals are in digital form (data reduction aspects thereof based on psychoacoustics G10L19/02)}
- G06N 3/045: Combinations of networks
- G10H 2210/056: for extraction or identification of individual instrumental parts, e.g. melody, chords, bass; Identification or separation of instrumental parts by their characteristic voices or timbres
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10H 2210/081: for automatic key or tonality recognition, e.g. using musical rules or a knowledge base
- G10H 2210/125: Medley, i.e. linking parts of different musical pieces in one single piece, e.g. sound collage, DJ mix
- G10H 2210/155: Musical effects
- G10H 2210/241: Scratch effects, i.e. emulating playback velocity or pitch manipulation effects normally obtained by a disc-jockey manually rotating a LP record forward and backward
- G10H 2210/325: Musical pitch modification
- G10H 2210/391: Automatic tempo adjustment, correction or control
- G10H 2220/101: for graphical creation, edition or control of musical data or parameters
- G10H 2220/106: using icons, e.g. selecting, moving or linking icons, on-screen symbols, screen regions or segments representing musical elements or parameters
- G10H 2230/015: PDA [personal digital assistant] or palmtop computing devices used for musical purposes, e.g. portable music players, tablet computers, e-readers or smart phones in which mobile telephony functions need not be used
- G10H 2240/325: Synchronizing two or more audio tracks or files according to musical features or musical timings
- G10H 2250/031: Spectrum envelope processing
- G10H 2250/035: Crossfade, i.e. time domain amplitude envelope control of the transition between musical sounds or melodies, obtained for musical purposes, e.g. for ADSR tone generation, articulations, medley, remix
- G10H 2250/311: Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
- G10H 2250/641: Waveform sampler, i.e. music samplers; Sampled music loop processing, wherein a loop is a sample of a performance that has been edited to repeat seamlessly without clicks or artifacts
- G10L 21/0272: Voice signal separating
- H04B 1/1646: {adapted for the reception of stereophonic signals}
- H04H 60/04: Studio equipment; Interconnection of studios
- H04R 5/04: Circuit arrangements, {e.g. for selective connection of amplifier inputs/outputs to loudspeakers, for loudspeaker detection, or for adaptation of settings to personal preferences or hearing impairments (combinations of amplifiers H03F3/68; stereophonic systems H04S)}
- H04R 27/00: Public address systems (circuits for preventing acoustic reaction H04R3/02; circuits for distributing signals to loudspeakers H04R3/12; {monitoring or testing arrangements for public address systems H04R29/007}; amplifiers H03F)
- H04R 2227/003: Digital PA systems using, e.g. LAN or internet
- H04R 2227/005: Audio distribution systems for home, i.e. multi-room use
- H04R 2420/01: Input selection or mixing for amplifiers or loudspeakers
- H04R 2430/01: Aspects of volume control, not necessarily automatic, in sound systems
- H04R 2430/03: Synergistic effects of band splitting and sub-band processing
- H04S 2400/13: Aspects of volume control, not necessarily automatic, in stereophonic sound systems
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04S 2420/07: Synergistic effects of band splitting and sub-band processing
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)
- H04H: BROADCAST COMMUNICATION (multiplex communication H04J; pictorial communication aspects of broadcast systems H04N)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04S: STEREOPHONIC SYSTEMS
- H04B: TRANSMISSION

## Abstract

The present invention relates to a method for processing and playing audio data comprising the steps of receiving mixed input data and playing recombined output data. Furthermore, the invention relates to a device for processing and playing audio data, preferably DJ equipment, comprising an audio input unit for receiving a mixed input signal, a recombination unit and a playing unit for playing recombined output data. In addition, the present invention relates to a method and a device for representing audio data, i.e. on a display.

## Claims

### Claim 1

A method for processing and playing audio data, comprising the steps of:
receiving mixed input data, said mixed input data being a sum signal obtained from mixing at least one first source track with at least one second source track;
decomposing the mixed input data to obtain at least a first decomposed track resembling the at least one first source track;
generating output data based on the first decomposed track; and
playing the output data through an audio output.

### Claim 2 (depends on 1)

The method of claim 1, further comprising the following steps:
reading a control input from a user, said control input representing a desired setting of a first volume level of the first decomposed track and a second volume level of a second track;
recombining at least the first decomposed track at the first volume level with the second track at the second volume level to generate recombined output data; and
playing the recombined output data.

### Claim 3 (depends on 2)

The method of claim 2, wherein the second track is obtained in the step of decomposing the mixed input data and forms a second decomposed track resembling the at least one second source track.

### Claim 4 (depends on 1)

The method of claim 1, wherein decomposing the mixed input data is carried out segment-wise;
wherein decomposing the mixed input data is carried out based on a first segment of the mixed input data such as to obtain a first segment of output data; and
wherein decomposing the mixed input data of a second segment of the mixed input data is carried out while playing the first segment of output data.

### Claim 5 (depends on 1)

The method of claim 1, wherein the receiving, decomposing, generating and playing steps are carried out in a continuous process.

### Claim 6 (depends on 1)

The method of claim 1, wherein the mixed input data are received via streaming from a remote server.

### Claim 7 (depends on 1)

The method of claim 1,
wherein receiving the mixed audio data comprises:
receiving an input audio file having a predetermined file size and a predetermined playback duration is received, wherein the input audio file contains first audio data to play the mixed input data, and
extracting a first segment of the input audio file, wherein the first segment of the input audio file contains second audio data to play the mixed input data within a first time interval smaller than the predetermined playback duration;
wherein decomposing the mixed input data to obtain at least a first decomposed track comprises decomposing the first segment of the input audio file to obtain a first segment of the first decomposed track;
wherein generating output data based on the first decomposed track comprises generating a first segment of the output data from the first segment of the first decomposed track; and
wherein the method further comprises the steps of:
extracting a second segment of the input audio file, which is different from the first segment of the unput audio file and which contains third audio data to play the mixed input data within a second time interval smaller than the predetermined playback duration of the input audio file and shifted in time with respect to the first time interval; and
decomposing the second segment of the input audio file to obtain a second segment of the first decomposed track.

### Claim 8 (depends on 4)

The method of claim 4, wherein the size of the first segment of the input audio file or the length of the first time interval is set such that the time required for decomposing the first segment of the input audio file is smaller than 2 seconds.

### Claim 9 (depends on 1)

The method of claim 1, further comprising the steps of:
receiving an input audio file having a predetermined file size and a predetermined playback duration, which contains first audio data to play the mixed input data;
partitioning the input audio file into a plurality of consecutive segments of the input audio file, wherein the segments of the input audio file contain audio data to play the mixed input data within a plurality of consecutive time intervals;
receiving a play position command from a user representing a user's command to play the input audio file from a certain start play position;
identifying a first segment of the input audio file of the plurality of segments of the input audio file such that the start play position is within the time interval which corresponds to the first segment of the input audio file;
decomposing the first segment of the input audio file to obtain a first segment of the first decomposed track;
generating a first segment of the output data based on the first segment of the first decomposed track; and
playing the first segment of the output data starting at the start play position, wherein the play position is later than or equal to the start of the time interval of the first segment of the output data.

### Claim 10 (depends on 1)

The method of claim 1, wherein the mixed input data are first mixed input data being a sum signal obtained from mixing at least a first source track with a second source track and wherein the method further comprises the steps of:
receiving second mixed input data, said second mixed input data being a sum signal obtained from mixing at least one third source track with at least one fourth source track;
decomposing the second mixed input data to obtain a third decomposed track resembling the at least one third source track, and a fourth decomposed track resembling the at least one fourth source track;
wherein said control input represents a desired setting of the first volume level of the first decomposed track, the second volume level of the second decomposed track, a third volume level of the third decomposed track, and a fourth volume level of the fourth decomposed track; and
wherein the recombined output data is generated by recombining the first decomposed track at the first volume level, the second decomposed track at the second volume level, the third decomposed track at the third volume level and the fourth decomposed track at the fourth volume level.

### Claim 11 (depends on 1)

The method of claim 1, wherein at least one of the mixed input data or the decomposed track signals represent stereo signals, each comprising a left-channel signal portion and a right-channel signal portion, respectively.

### Claim 12 (depends on 1)

The method of claim 1, wherein decomposing the mixed input data includes processing the mixed input data by an AI system, said AI system trained by a plurality of sets of training audio data, wherein each set of training audio data at least includes a first source track and a mixed track being a sum signal obtained from mixing at least the first source track or a track that resembles the first source track, with a second source track.

### Claim 13 (depends on 1)

The method of claim 1, wherein the mixed input data are processed within a first AI system and a second AI system separate from the first AI system, wherein the first AI system processes the mixed input data to obtain only the first decomposed track and the second AI system processes the mixed input data to obtain only a second decomposed track.

### Claim 14 (depends on 1)

The method of claim 1, wherein said mixed input data are first mixed input data based on a periodic beat structure and wherein the method further comprises:
receiving second mixed input data different from the first mixed input data and having a periodic beat signal; and
performing at least one of a tempo matching processing or a key matching processing,
wherein the tempo matching processing comprises: receiving first input data obtained from the first mixed input data and second input data obtained from the second mixed input data, time stretching or resembling of at least one of the first input data and the second input data, and outputting first output data and second output data which have mutually matching tempos, and
wherein the key matching processing comprises: receiving the first input data obtained from the first mixed input data and the second input data obtained from the second mixed input data, pitch shifting of at least one of the first input data or the second input audio data, and outputting the first output data and the second output data which have mutually matching keys.

### Claim 15

A device for processing and playing audio data comprising:
an audio input unit for receiving mixed input data, said mixed input data being a sum signal obtained from mixing at least a first source track with at least a second source track;
a decomposing unit connected to the audio input unit for decomposing the mixed input data to obtain at least a first decomposed track resembling the first source track; and
a playing unit for playing output data based on the first decomposed track.

### Claim 16 (depends on 15)

The device of claim 15, further comprising a recombination unit (32) for recombining at least the first decomposed track with a second track to generate the output data for the playing unit.

### Claim 17 (depends on 15)

The device of claim 15, further comprising a recompose controlling section adapted to be controlled by a user to generate a control input representing a desired setting of a first volume level of the first decomposed track and a second volume level of the second track, wherein the recombination unit is configured to recombine at least the first decomposed track at the first volume level with the second track at the second volume level to generate the output data.

### Claim 18 (depends on 17)

The device of claim 17, wherein the audio input unit is a first audio input unit for receiving first mixed input data being a sum signal obtained from mixing at least a first source track with a second source track, and the decomposing unit is a first decomposing unit;
wherein the device further comprises: a second audio input unit for receiving second mixed input data, said second mixed input data being a sum signal obtained from mixing at least a third source track with a fourth source track, and a second decomposing unit connected to the second audio input unit for decomposing the second mixed input data to obtain a third decomposed track resembling the third source track and a fourth decomposed track resembling the fourth source track;
wherein the recompose controlling section is adapted to be controlled by a user to generate a control input representing a desired setting of the first volume level of the first decomposed track, the second volume level of the second decomposed track, a third volume level of the third decomposed track, and a fourth volume level of the fourth decomposed track; and
wherein the recombination unit is adapted to generate the recombined output data by recombining the first decomposed track at the first volume level, the second decomposed track at the second volume level, the third decomposed track at the third volume level and the fourth decomposed track at the fourth volume level.

### Claim 19 (depends on 17)

The device of claim 17, wherein the recompose controlling section comprises at least one recompose control element which is operable by a user in a single control operation for controlling the first volume level and the second volume level, while at the same time decreasing the other one of the first volume level and the second volume level.

### Claim 20 (depends on 17)

The device of claim 17, wherein the recompose controlling section comprises:
a first recompose control element which is operable by a user in a single control operation for controlling the first volume level and the second volume level while at the same time decreasing the other one of the first volume level and the second volume level; and
a mix control element, which is operable by the user in a single control operation for controlling a first sum signal and a second sum signal, while at the same time decreasing the other one of the first sum signal and the second sum signal, the first sum signal being a sum of the first decomposed track at the first volume level and the second decomposed track at the second volume level and the second sum signal being a sum of the third decomposed track at the third volume level and the fourth decomposed track at the fourth volume level.

### Claim 21 (depends on 15)

The device of claim 15, further comprising:
an input audio file buffer for loading therein segments of an input audio file having a predetermined file size and a predetermined playback duration, wherein input audio file contains audio data to play the mixed input data;
a first segment buffer connected to the decomposing unit to receive and store a segment of the first decomposed track obtained from a segment of the input audio file;
a second segment buffer connected to the decomposing unit to receive and store a segment of the second decomposed track obtained from the segment of the input audio file;
wherein the playing unit comprises an audio interface having an analog-to-digital converter to generate an analog audio signal from the recombined output data, said audio interface having an audio buffer for buffering portions of the output data for playing; and
wherein the size of at least one of the first segment buffer or the second segment buffer is larger than the size of the audio buffer of the audio interface, but smaller than the input audio file data or the predetermined file size of the input audio file.

### Claim 22 (depends on 15)

The device of claim 15, wherein the audio input unit is a first audio input unit for receiving first mixed input data based on a periodic beat structure, and wherein the device further comprises:
a second audio input unit for receiving second mixed input data (B) different from the first mixed input data (A) and based on a periodic beat signal; and
at least one of a tempo matching unit or a key matching unit, and
wherein the tempo matching unit is arranged to receive a first input data obtained from the first mixed input data and second input data obtained from the second mixed input data, and wherein the tempo matching unit comprises a time stretching unit adapted to time stretch at least one of the first input data and the second input audio data, and to output first output data and second output data which have mutually matching tempos, and
wherein the key matching unit is arranged to receive the first input data obtained from the first mixed input data and the second input data obtained from the second mixed input data, and wherein the key matching unit comprises a pitch shifting unit adapted to pitch shift at least one of the first input data and the second input audio data, and to output the first output data and the second output data which have mutually matching keys.

### Claim 23 (depends on 15)

The device of claim 15, wherein the audio input unit, the decomposing unit and the playing unit are integrated within a single unit.

### Claim 24

A method for representing audio data, said audio data comprising at least a first track and a second track, wherein the first track and the second track are components of a joint audio mix, and wherein the method comprises displaying a first waveform representative of the first track and displaying a second waveform representative of the second track, wherein the first waveform and the second waveform are displayed in an overlaid manner using one single baseline, and wherein the the first waveform and the second waveform are displayed using at least one of different signal axes or different drawing styles such as to be visually distinguishable from one another.

### Claim 25 (depends on 24)

The method of claim 24, wherein the first waveform is displayed using a first drawing style which draws signal portions of the first waveform primarily or exclusively in a positive region relative to a baseline, and wherein the second waveform is displayed using a second drawing style which draws signal portions of the second waveform primarily or exclusively in a negative region relative to the same baseline.

### Claim 26 (depends on 24)

The method of claim 24, wherein the first and second waveforms are displayed using first and second drawing styles which both draw primarily or exclusively the positive signal portion or which both draw primarily or exclusively the negative signal portion, and wherein the first waveform is displayed using a first signal axis and the second waveform is displayed using a second signal axis that runs opposite to the first signal axis.

### Claim 27 (depends on 24)

The method of claim 24, wherein at least one of the first waveform or the second waveform is displayed by rendering the at least one of the first waveform or the second waveform in a predetermined time interval with a color that depends on a frequency information of the respective track within the predetermined time interval.

### Claim 28 (depends on 24)

The method of claim 24, further comprising:
receiving mixed input data, said mixed input data being a sum signal obtained from mixing at least one first source track with at least one second source track;
decomposing the mixed input data to obtain at least a first decomposed track resembling the at least one first source track, and a second decomposed track resembling the at least one second source track;
reading a control input from a user, said control input representing a desired setting of a first volume level of the first decomposed track and a second volume level of the second decomposed track;
displaying a first waveform representative of the first decomposed track and displaying a second waveform representative of the second decomposed track, wherein the first waveform and the second waveform are displayed in an overlaid manner using one single baseline, and wherein the waveforms are displayed using at least one of different signal axes or different drawing styles such as to be visually distinguishable from one another; and
wherein the first waveform is displayed with its signal axis being scaled or its appearance being modified depending on the first volume level, wherein the second waveform is displayed with its signal axis being scaled or its appearance being modified depending on the second volume level.

### Claim 29 (depends on 28)

The method of claim 28, wherein the first waveform and second waveform are displayed with their signal axes being scaled on the basis of current values of the first volume level and the second volume volume level within a time period of not more than 2 seconds.

### Claim 30

A device for processing and playing audio data comprising:
a processing unit for processing audio data of at least a first track and a second track;
a controlling section adapted to be controlled by a user to generate a control input representing a desired setting of a first volume level of the first track and a second volume level of the second track;
a recombination/mixing unit configured to combine the first track at a first volume level with the second track at a second volume level to generate output data;
a visualization unit configured to generate waveform data for visualizing at least one waveform based on the first track, the second track and the control input; and
a playing unit for playing audio data based on the output data.

### Claim 31 (depends on 30)

The device of claim 30, wherein the visualization unit is configured to generate at least one of a first waveform based on the first track or a second waveform based on the second track, wherein a scaling of a signal axis or a drawing style of the first waveform is set depending on the first volume level, and, wherein a scaling of a signal axis or a drawing style of the second waveform is set depending on the second volume level.

### Claim 32 (depends on 30)

The device of claim 30, wherein the visualization unit is configured to calculate a combination track representing a combination of at least the first track at the first volume level and the second track at the second volume level, and to generate the waveform data such as to visualize the waveform of the combination track.

### Claim 33 (depends on 30)

The device of claim 30, wherein the device is configured to allow the waveform data based on a particular control input to be generated and displayed on the vizualization unit within a time period of not more than 2 seconds after the particular control input is generated by the user.

### Claim 34 (depends on 30)

The device of claim 30, wherein the visualization unit is configured to generate the waveform data for visualizing a first waveform based on the first track and the control input, and a second waveform based on the second track and the control input, and wherein the waveform data are generated such as to display the first waveform and the second waveform in an overlaid manner using one baseline, but at least one of different signal axes or different drawing styles such as to be visually distinguishable from one another.

### Claim 35 (depends on 30)

The device claim 30, wherein the first track is the first decomposed track and the second track is the second decomposed track.

### Claim 36

A device for processing and playing audio data, comprising:
an audio input unit for receiving a first track and a second track, said first track being a component of an audio mix track;
a controlling section adapted to be controlled by a user to generate a control input representing a desired setting of a first volume level of the first track and a second volume level of the second track;
a playing unit for playing output data base on the first track at the first volume level and the second track at the second volume level; and
wherein the controlling section comprises at least one control element which is operable by a user in a single control operation for controlling the first volume level and the second volume level, in particular increasing one of the first volume level and the second volume level, while at the same time decreasing the other one of the first volume level and the second volume level.

### Claim 37 (depends on 36)

The device of claim 36, wherein the first track and the second track are components of the same audio mix track.

### Claim 38

A device for processing and playing audio data, comprising:
an audio input unit for receiving mixed input data, said mixed input data being a sum signal obtained from mixing a vocal component, a drum component and a harmonic component;
a decomposing unit connected to the audio input unit for decomposing the mixed input data to obtain a first decomposed track resembling the vocal component, a second decomposed track resembling the drum component, and a third decomposed track resembling the harmonic component;
a recompose controlling section adapted to be controlled by a user and including a first control element for controlling a first volume level of the first decomposed track, a second control element for controlling a second volume level of the second decomposed track, and a third control element for controlling a third volume level of the third decomposed track;
a recombination unit adapted to generate output data by recombining the first decomposed track at the first volume level, the second decomposed track at the second volume level, and the third decomposed track at the third volume level; and
a playing unit for playing the output data.