# US 2004/0069123 A1 · Automatic recognition and matching of tempo and phase of pieces of music, and an interactive music player based thereon

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2004/0069123 A1
- Document ID: US20040069123A1
- Status: Published application
- Kind: A1
- Application number: 10/251,000
- Filed: 2003-07-08
- Published: 2004-04-15
- Earliest claimed: 2001-01-13
- Assignees: Native Instruments Software Synthesis GmbH
- Inventors: Friedmann Becker; Thomas Holl; Michael Kurz; Toine Diepstraten; Daniel Haver
- Web page: https://patentgrep.com/patent/US20040069123A1

## CPC

- G10H 1/00: Details of electrophonic musical instruments
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- G10H 2240/061: MP3, i.e. MPEG-1 or MPEG-2 Audio Layer III, lossy audio compression
- G10H 2240/325: Synchronizing two or more audio tracks or files according to musical features or musical timings
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE

## Abstract

A method of matching the tempo and phase in pieces of music which allows the conjunction of the pieces of music to form a continuous stream of music. The interactive music player which digitally executes the method of matching the tempo and phase in pieces of music is also disclosed.

## Claims

### Claim 1

Method for detecting the tempo and phase of a piece of music available in digital format with the following procedural stages:
approximation of the tempo (A) of the piece of music by means of a statistical evaluation (STAT) of the time intervals (Ti) between rhythm-relevant beat information in the digital audio data (Ei);
approximation of the phase (P) of the piece of music with reference to the position of the beats in the digital audio data within the time grid of a reference oscillator (MCLK) oscillating at a frequency proportional to the tempo established;
successive correction of the established tempo (A) and phase (P) of the piece of music with reference to a possible phase shift of the reference oscillator (MCLK) relative to the digital audio data by evaluating the resulting systematic phase shift and regulating the frequency of the reference oscillator in proportion to the phase shift established.

### Claim 2 (depends on 1)

Method for detecting the tempo and phase of a piece of music available in digital format according to claim 1, wherein rhythm-relevant information (Ti) is obtained by band-pass filtering (F1, F2) of the basic digital audio data in various frequency ranges.

### Claim 3 (depends on 1, 2)

Method for detecting the tempo and phase of a piece of music available in digital format according to claim 1 or 2, wherein rhythmic intervals in the audio data are transformed, if necessary by raising their frequency by a power of 2, into a pre-defined frequency octave (OKT), where they provide time intervals (T1io . . . T3io) for establishing the tempo.

### Claim 4 (depends on 3)

Method for detecting the tempo and phase of a piece of music available in digital format according to claim 3, wherein the frequency transformation (OKT) is preceded by a grouping of rhythmic intervals (Ti), especially into pairs (T2i) or groups of three (T3i), by addition of their time values.

### Claim 5 (depends on 2, 3, 4)

Method for detecting the tempo and phase of a piece of music available in digital format according to one or more of the preceding claims 2 to 4, wherein the quantity of data obtained for time intervals (BPM1, BPM2) in the rhythm-relevant beat information (N) is investigated for accumulation points, and the approximation of tempo is carried out on the basis of information referring to an accumulation maximum.

### Claim 6

Method for detecting the tempo and phase of piece of music available in digital format according to one or more of the preceding claims, wherein, for the approximation of the phase (P) of the piece of music, the phase of the reference oscillator (MCLK) is selected in such a manner that the maximum possible agreement is set between the rhythm-relevant beat information in the digital audio data and the zero passes of the reference oscillator (MCLK).

### Claim 7

Method for detecting the tempo and phase of a piece of music available in digital format according to any one of the preceding claims, wherein a successive correction (2,3,4,5) of the established tempo and phase of the piece of music takes place at regular intervals over such short time intervals that resulting correction movements and/or correction shifts remain below the threshold of audibility.

### Claim 8

Method for detecting the tempo and phase of a piece of music available in digital format according to one or more of the preceding claims, wherein all successive corrections of the established tempo and phase of the piece of music are accumulated (4) over time and further corrections are made on this basis with constantly increasing precision, wherein, in particular, successive corrections are continued until the error in the established tempo falls below a predetermined tolerable error threshold value, especially an error threshold value of less than 0.1%.

### Claim 9

Method for detecting the tempo and phase of a piece of music available in digital format according to one or more of the preceding claims, wherein, if the corrections over a predetermined period of time (6) are always either negative or positive, a new (RESET) approximation of tempo (A) and phase (P) is implemented with subsequent successive correction (2,3,4,5).

### Claim 10

Method for synchronising at least two pieces of music available in digital format with the following procedural stages:
complete establishment of the tempo and phase of the first piece of music according to any one of the preceding claims,
approximation of the tempo and phase of the other piece of music according to any one of the preceding claims,
matching of the playback rate and the playback phase of this other piece of music by successive adaptation of the frequency and phase of the reference oscillator allocated to this other piece of music to the frequency and phase of the reference oscillator allocated to the first piece of music.

### Claim 11 (depends on 10)

Method for synchronising at least two pieces of music available in digital format according to claim 10, wherein the playback rate and playback phase of the other piece of music is matched on the basis of a possible phase shift in the reference oscillator allocated to this other piece of music relative to the reference oscillator of the first piece of music, and the resulting systematic phase shift is evaluated and the frequency of the reference oscillator allocated to the other piece of music is regulated in proportion to the phase shift established.

### Claim 12 (depends on 10, 11)

Music player, wherein at least two pieces of music available in digital format can be synchronised in real-time according to claim 10 or 11, especially wherein rhythm-relevant beat information (Ti) from a predetermined past time relative to a current playing position of the piece of music are used, in each case, as a basis for establishing the tempo in real-time.

### Claim 13 (depends on 11, 12)

Music player according to any one of the preceding claims 11 or 12, wherein synchronised pieces of music can be automatically sorted and played back as a complete work with a unified rhythm.

### Claim 14 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13)

Interactive music player, especially according to any one of the preceding claims 11 to 13, which provides
a means for a graphic, real-time representation of beat thresholds, established with a tempo and phase detector function, especially a tempo and phase detector function according to any one of claims 1 to 10, during playback of a piece of music,
a first control element (R1) for switching between a first operating mode (a), in which the playback position and/or the playback rate can be directly influenced by the user in real-time, and
a second control element (R2) for manipulating the playback position in real-time.

### Claim 15 (depends on 14)

Interactive music player according to claim 14, with
a means for graphic representation of the current playback position, with which an amplitude-envelope-curve of the sound wave form of the piece of music being played can be represented over a predetermined time before and after the current playback position, wherein the representation moves in real-time at the playback tempo of the piece of music, and with
a means for smoothing (LP, SL) a stepped sequence of time-limited, playback-position-data, predetermined by means of the second control element (R2), to form an evenly changing signal with a time resolution corresponding to the audio sampling rate,
wherein a means is provided, especially for smoothing a stepped sequence of time-limited, playback-position-data, for ramp smoothing (SL), by means of which, with every specified playback-position message, a ramp with a constant gradient can be resolved, which, over a predetermined time interval, changes the smoothed signal from its previous value to the value of the playback-position message.

### Claim 16 (depends on 15)

Interactive music player according to claim 15, wherein a linear digital low-pass filter (LP), especially a second order resonance filter, is used to smooth a stepped sequence of time-limited, predetermined playback-position-data.

### Claim 17 (depends on 12, 13, 14, 15, 16)

Interactive music player according to any one of the preceding claims 12 to 16, wherein, in the case of a change between the operating modes (a,b), the position reached in the preceding mode is used as the starting position in the new mode.

### Claim 18 (depends on 14, 15, 16, 17)

Interactive music player according to any one of the preceding claims 14 to 17, wherein, in the event of a change between the operating modes (a,b), the current playback rate (DIFF) reached in the preceding mode can be changed by a smoothing function, especially a ramp smoothing function (SL) or a linear digital low-pass filter (LP), to the playback rate corresponding to the new operating mode.

### Claim 19 (depends on 12, 13, 14, 15, 16, 17, 18)

Interactive music player according to any one of the preceding claims 12 to 18, wherein an audio signal runs through a Scratch-Audio-Filter, in which the audio signal is subjected to pre-emphasis-filtering (PEF) and stored in a buffer (B), from which it can be read out (R) at a variable tempo in dependence upon the relevant playback rate, after which it is subjected to de-emphasis-filtering (DEF) and played back.

### Claim 20 (depends on 12, 13, 14, 15, 16, 17, 18, 19)

Interactive music player according to any one of the preceding claims 12 to 19, wherein, for one or more of the synchronised pieces of music, the length of a playback loop extending over one or more beats of the piece of music can be defined and played back in real-time in a beat-synchronised manner, on the basis of the tempo information established for the relevant piece of music.

### Claim 21 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20)

Interactive music player according to any one of the preceding claims 12 to 20, wherein, for one or more of the synchronised pieces of music, beat-synchronised jump marks can be defined in real-time and moved by whole-number multiples of beats within this piece of music on the basis of the phase information established for the relevant piece of music,
wherein each audio-data stream played back can be manipulated in real-time by signal processing means, especially by filtering devices and/or audio effects.

### Claim 22 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21)

Interactive music player according to any one of the preceding claims 12 to 21, wherein real-time interventions over the time sequence can be stored as digital control information (MIX_DATA), especially those relating to a mixing procedure for several pieces of music and/or additional signal processing.

### Claim 23 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22)

Interactive music player according to any one of the preceding claims 12 to 22, wherein mixing procedures for pieces of music and/or interactive interventions in pieces of music using audio-signal processing means can be stored, independently of digital audio information for the pieces of music, in the form of digital control information (MIX_DATA), especially for the purpose of reproduction as a new complete work.

### Claim 24 (depends on 22, 23)

Interactive music player according to any one of the preceding claims 22 or 23, wherein the stored digital control information has a format, which provides information for identifying the pieces of music processed and the time sequence of playback positions and status information for the control elements of the music player allocated to each of the pieces of music.

### Claim 25 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24)

Interactive music player according to any one of the preceding claims 12 to 24, which is realised with an appropriately programmed computer system fitted with audio interfaces.

### Claim 26 (depends on 11, 12)

Method for providing digital audio data in real-time for at least two pieces of music from a data source (CD-ROM) with only one reader unit, especially for synchronisation according to any one of claims 11 or 12, wherein the data source supplies audio data at a faster reading rate than the relevant playback rate, in that a relevant buffer memory (P1 . . . Pn), especially a ring buffer memory, is provided for each piece of music to be played-back (TR1 . . . TRn), and that the faster reading rate is used in order to fill the relevant buffer memory (P1 . . . Pn) with associated audio data in such a manner, that audio data are always available in time before and after a current playback position (A1 . . . An) of the relevant piece of music.

### Claim 27 (depends on 26)

Method for providing digital audio data according to claim 26, wherein the status of each buffer memory (P1 . . . Pn) is monitored to determine whether sufficient data are available, and when the level falls below a predetermined threshold value, a central instance (S), which is not coupled to the playback of the pieces of music (TR1 . . . TRn), is ordered to provide the necessary audio data and automatically requests the required regions of audio data from the data source (CD-ROM) and fills up the associated buffer memory (P1 . . . Pn) with the data obtained, wherein data which are no longer needed are over-written, especially during the filling up of a buffer memory (P1 . . . Pn).

### Claim 28 (depends on 26, 27)

Method for providing digital audio data according to claim 26 or 27, wherein the central instance (S) places requests received in parallel into an order to be worked through sequentially.

### Claim 29 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28)

Interactive music player according to any one of the preceding claims 12 to 25, wherein a CD-ROM drive (CD-ROM) operated according to any one of the preceding claims 26 to 28 is used as the data source for the pieces of music (TR1 . . . TRn).

### Claim 30 (depends on 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 25, 26, 27)

Computer software product, which can be loaded directly into the internal memory of a digital computer and comprises software sections, with which the procedural stages according to any one of claims 1 to 12 or 25 to 27 can be implemented when the software product is run on a computer.

### Claim 31 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25)

Data medium (D), especially a Compact Disk, which comprises
a first data region (D1) with digital audio data (AUDIO_DATA) from one or more pieces of music (TR1 . . . TRn) and
a second data region (D2) with a control file (MIX_DATA) with digital control information for controlling a music player, especially a music player according to any one of claims 12 to 25, wherein
the control data (MIX_DATA) in the second data region (D2) refer to audio data (AUDIO DATA) in the first data region (D1).

### Claim 32 (depends on 31)

Data medium (D) according to claim 31, wherein the digital control data (MIX_DATA) in the second data region (D2) represent mixing procedures for pieces of music and/or interactive interventions into pieces of music using audio signal processing means to provide a new complete work with the digital audio information (AUDIO_DATA) from pieces of music in the first data region (D1).

### Claim 33 (depends on 31, 32)

Data medium (D) according to claim 31 or 32, wherein stored digital control data (MIX_DATA) in the second data region (D2) have a format which provides information for the identifying the processed pieces of music (TR1 . . . TRn) in the first data region (D1) and a relevant time sequence of playback positions and status information for the control elements of the music player allocated to the processed pieces of music.

### Claim 34 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 31, 32, 33)

Computer software product (PRG_DATA), which is disposed on a data medium (D) according to any one of claims 31 to 33, which can be directly loaded into the internal memory of a digital computer and which comprises software sections, with which this digital computer assumes the function of a music player, especially a music player according to any one of claims 12 to 25 or 33, with which, in dependence upon the control data (MIX_DATA) in the second data region (D2) of the data medium (D2), which refer to audio data (AUDIO_DATA) in the first data region (D1) of the data medium (D), a complete work represented by the control data (MIX_DATA) can be played back, when the software product (PRG_DATA) is run on the computer.

### Claim 35 (depends on 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25)

Data structure, which is preferably stored in a control file (MIX_DATA), with digital control information for controlling a music player, especially a music player according to any one of claims 12 to 25, wherein the control data (MIX_DATA) refer to audio data (AUDIO_DATA) in such a manner that a mixing of audio data is possible, in that a time sequence of control data, a time sequence of exact playback positions in the audio source and intervals with complete status information for all control elements providing new starting points for playback, are preferably stored in the data structure.
