# US 2011/0255700 A1 · Detecting Musical Structures

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2011/0255700 A1
- Document ID: US20110255700A1
- Status: Published application
- Kind: A1
- Application number: 12/760,522
- Filed: 2010-04-14
- Published: 2011-10-20
- Earliest claimed: 2010-04-14
- Assignees: Apple Inc.
- Inventors: Cynthia Maxwell; Frank Martin Ludwig Gunter Baumgarte
- Web page: https://patentgrep.com/patent/US20110255700A1

## CPC

- H04R 5/027: Spatial or constructional arrangements of microphones, e.g. in dummy heads
- G10H 1/368: {displaying animated or moving pictures synchronized with the music or audio part}
- G10H 1/40: Rhythm
- G10H 2210/076: for extraction of timing, tempo; Beat detection
- H04S 2400/15: Aspects of sound capture and related signal processing for recording or reproduction
- H04R: LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS (producing sounds with frequency not determined by supply frequency G10K)
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
- H04S: STEREOPHONIC SYSTEMS

## Abstract

Among other things, techniques and systems are disclosed for detecting musical structures, such as downbeats. In one aspect, a method performed by a data processing device includes receiving an input audio signal. The method includes detecting a meter in the received audio signal. Detecting the meter includes generating an envelope of the received audio signal; generating an autocorrelation phase matrix having a two-dimensional array based on the generated envelope to identify a dominant periodicity in the received audio signal; and filtering both dimensions of the generated autocorrelation phase matrix to enhance peaks in the two-dimensional array. The meter represents a time signature of the input audio signal having multiple beats. Additionally, the method includes identifying a downbeat as a first beat in the detected meter.

## Claims

### Claim 1

A method performed by a data processing device, the method comprising:
receiving an input audio signal;
detecting a meter in the received audio signal, detecting the meter comprising
generating an envelope of the received audio signal,
generating an autocorrelation phase matrix having a two-dimensional array based on the generated envelope to identify a dominant periodicity in the received audio signal, and
filtering both dimensions of the generated autocorrelation phase matrix to enhance peaks in the two-dimensional array,
wherein the meter represents a time signature of the input audio signal having multiple beats; and
identifying a downbeat as a first beat in the detected meter.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the envelope comprises:
generating an analytic signal based on the received input audio signal.

### Claim 3 (depends on 1)

The method of claim 1, wherein detecting the meter further comprises:
downsampling the generated envelope to reduce a complexity of the estimated envelope.

### Claim 4 (depends on 1)

The method of claim 1, wherein detecting the meter further comprises:
determining a correlation between the generated envelope and a time shifted version of the generated envelope,
wherein the time shifted version is shifted in time by a time lag.

### Claim 5 (depends on 4)

The method of claim 4, wherein the time lag represents an integer multiple of a beat rate of the received input audio signal.

### Claim 6 (depends on 4)

The method of claim 4, wherein generating the autocorrelation phase matrix comprises:
computing the autocorrelation phase matrix having the two-dimensional array based on the determined correlation, wherein a first dimension of the two-dimensional array is associated with the time lag and a second dimension of the two-dimensional array is associated with a phase shift between the generated envelope and the time shifted version.

### Claim 7 (depends on 6)

The method of claim 6, wherein computing the autocorrelation phase matrix comprises:
varying a length of the time lag in the first dimension; and
varying a size of the phase shift in the second dimension.

### Claim 8 (depends on 6)

The method of claim 6, wherein detecting the meter further comprises:
generating an enlarged autocorrelation phase matrix by extending the filtered autocorrelation phase matrix in the second dimension to avoid a triangular shape in the autocorrelation phase matrix.

### Claim 9 (depends on 8)

The method of claim 8, wherein detecting the meter further comprises:
performing a circular autocorrelation operation on the generated enlarged autocorrelation phase matrix using an autocorrelation function.

### Claim 10 (depends on 9)

The method of claim 9, wherein detecting the meter further comprises:
generating a smoothed autocorrelation function that removes a variable offset from the autocorrelation function.

### Claim 11 (depends on 10)

The method of claim 10, wherein detecting the meter further comprises:
subtracting the generated smoothed autocorrelation function from the autocorrelation function;
removing a DC offset from a result of the subtracting; and
identifying peaks of the autocorrelation function.

### Claim 12 (depends on 11)

The method of claim 11, wherein detecting the meter in the received audio signal further comprises:
applying a weighting function to the autocorrelation function to reduce a number of false detection of peaks.

### Claim 13 (depends on 12)

The method of claim 12, wherein detecting the meter further comprises:
identifying a location of a highest peak from the detected peaks; and
removing remaining peaks from the autocorrelation function.

### Claim 14 (depends on 13)

The method of claim 13, wherein detecting the meter further comprises:
cleaning the autocorrelation function using a threshold value.

### Claim 15 (depends on 14)

The method of claim 14, wherein detecting the meter further comprises:
testing the autocorrelation function using multiple meter templates; and
responsive to the testing, identifying the meter in the received audio signal.

### Claim 16 (depends on 1)

The method of claim 1, wherein identifying a downbeat as a first beat in the detected meter comprises:
identifying a strongest beat from the multiple beats within the detected meter; and
comparing the identified strongest beat with neighboring beats to detect the downbeat as the first beat in the detected meter.

### Claim 17 (depends on 1)

The method of claim 1, wherein identifying a downbeat as a first beat in the detected meter comprises:
identifying a first beat from the multiple beats within the detected meter; and
comparing the identified first beat with neighboring beats to detect the downbeat as the first beat in the detected meter.

### Claim 18 (depends on 1)

The method of claim 1, comprising:
using the detected downbeat to synchronize the received audio signal with a video signal.

### Claim 19

A system comprising:
a user input unit to receive an input audio signal;
a meter detection unit to deconstruct the received input audio signal to detect at least one temporal location associated with a change in the input audio signal, wherein the temporal location comprises a meter that contains multiple beats;
a downbeat detection unit to:
identify a downbeat as a first beat in the detected meter, and
identify boundaries of the received input audio signal based on the detected downbeat; and
a data compression unit to:
receive the identified boundaries from the downbeat detection unit, and
perform data compression using the identified boundaries as markers for compressing data.

### Claim 20

A data processing device comprising:
a digital signal processing unit to detect downbeats in an audio signal, the digital signal processing unit comprising:
a meter detection unit to detect a meter in the received audio signal, wherein the meter comprises multiple beats; and
a downbeat detection unit to
identify a downbeat as a first beat in the detected meter, and
identify boundaries of the received audio signal based on the detected downbeat,
wherein the digital signal processing unit is configured to use the identified boundaries as triggers for executing one or more operations in the data processing device or a different device.

### Claim 21 (depends on 20)

The data processing device of claim 20, wherein the digital signal processing unit is configured to:
synchronizing the received audio signal with video data based on the identified boundaries.

### Claim 22 (depends on 20)

The data processing device of claim 20, wherein the digital signal processing unit is configured to:
realigning recorded audio data based on the identified boundaries.

### Claim 23 (depends on 20)

The data processing device of claim 20, wherein the digital signal processing unit is configured to:
mix two different audio data together by aligning the identified boundaries.

### Claim 24 (depends on 20)

The data processing device of claim 20, further comprising:
a data compression unit to perform data compression using the identified boundaries as markers for data compression.

### Claim 25

A non-transitory computer readable storage medium embodying instructions, which, when executed by a processor, cause the processor to perform operations comprising:
detecting a meter in the received audio signal, wherein the meter contains multiple beats;
identifying a downbeat as a first beat in the detected meter;
identifying boundaries of the received audio signal based on the detected downbeat; and
using the identified boundaries as markers for deconstructing the received input audio signals into multiple components.

### Claim 26 (depends on 25)

The non-transitory computer readable medium of claim 25, wherein using the identified boundaries as markers for deconstructing the received input audio signals into multiple components comprises:
compressing the input audio signal.

### Claim 27 (depends on 25)

The non-transitory computer readable medium of claim 25, wherein using the identified boundaries as markers for deconstructing the received input audio signals into multiple components comprises:
rearranging the input audio signal.

### Claim 28 (depends on 25)

The non-transitory computer readable medium of claim 25, wherein using the identified boundaries as markers for deconstructing the received input audio signals into multiple components comprises:
synchronizing the input audio signal with a video signal.
