# US 2008/0300702 A1 · Music similarity systems and methods using descriptors

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2008/0300702 A1
- Document ID: US20080300702A1
- Status: Published application
- Kind: A1
- Application number: 12/128,917
- Filed: 2008-05-29
- Published: 2008-12-04
- Earliest claimed: 2007-05-29
- Assignees: Universitat Pompeu Fabra
- Inventors: Emilia Gomez; Perfecto Herrera; Pedro Cano Vila; Jordi Janer; Joan Serra; Jordi Bonada; Shadi Walid El-Hajj; Thomas Etienne Aussenac; Gunnar Nils Holmberg
- Web page: https://patentgrep.com/patent/US20080300702A1

## CPC

- G10L 25/48: specially adapted for particular use
- G06F 16/683: using metadata automatically derived from the content
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Systems and methods for determining similarity between two or more audio pieces are disclosed. An illustrative method for determining musical similarities includes extracting one or more descriptors from each audio piece, generating a vector for each of the audio pieces, extracting one or more audio features from each of the audio pieces, calculating values for each audio feature, calculating a distance between a vector containing the normalized values and the vectors containing the audio pieces, and outputting a response to a user or process indicating the similarity between the audio pieces. The descriptors can be used in performing content-based audio classification and for determining similarities between music. The descriptors that can be extracted from each audio piece can include tonal descriptors, dissonance descriptors, rhythm descriptors, and spatial descriptors.

## Claims

### Claim 1

A method for determining similarity between two or more audio pieces, comprising:
extracting one or more descriptors from each of the audio pieces;
generating a vector for each of the audio pieces;
extracting one or more audio features from each of the audio pieces and storing the features in a database;
calculating values for each audio feature;
normalizing the values for each audio feature;
calculating a distance between a vector containing the normalized values and the vectors containing the audio pieces; and
outputting a result to a user or process.

### Claim 2 (depends on 1)

The method of claim 1, wherein the descriptors include one or more dissonance descriptors, tonal descriptors, rhythm descriptors, and/or spatial descriptors.

### Claim 3 (depends on 2)

The method of claim 2, wherein the dissonance descriptors include an audio piece dissonance descriptor, a dissonance of chords descriptor, and/or a spectral complexity descriptor.

### Claim 4 (depends on 3)

The method of claim 3, wherein, in extracting an audio piece dissonance descriptor from each audio piece, the method includes:
dividing a digitized audio piece into overlapping window frames;
eliminating or reducing noisy frequencies within the window frames;
performing a frequency quantization of each windowed frame and acquiring a vector;
weighting the vector;
extracting local spectral maximums for each frame;
computing a dissonance value based on extracted spectral maximum pairs; and
averaging the dissonances of all window frames to obtain the dissonance of the audio piece.

### Claim 5 (depends on 4)

The method of claim 4, wherein a spectral complexity is computed as a step or steps used in determining the dissonance of the audio piece.

### Claim 6 (depends on 3)

The method of claim 3, wherein, in extracting a dissonance of chords descriptor from each audio piece, the method includes:
determining the fundamentals and associated harmonics of two chords in an audio piece;
obtaining a spectrum of the frequencies corresponding to the fundamentals and their harmonics;
computing a dissonance of the spectrum;
weighting the spectrum;
computing the dissonance for two consecutive chords; and
averaging the sequence of dissonances from the two consecutive chords.

### Claim 7 (depends on 2)

The method of claim 2, wherein the tonal descriptors include a Harmonic Pitch Class Profile (HPCP) descriptor, a chord detection descriptor, a key detection descriptor, a local tonality detection descriptor, a cover versions detection descriptor, and/or a western music descriptor.

### Claim 8 (depends on 7)

The method of claim 7, wherein, in extracting a Harmonic Pitch Class Profile (HPCP) descriptor from each audio piece, the method includes computing an HPCP vector by:
dividing a digitized audio piece into windowed frames;
performing a frequency quantization of each windowed frame and acquiring a vector;
eliminating or reducing noisy frequencies within the windowed frames;
extracting local spectral maximums for each window frame;
determining a global tuning frequency value from the spectral maximums;
filtering one or more bands of spectral peaks;
performing a frequency mapping on the bands and obtaining two HPCP vectors;
weighting the frequency contribution and harmonics within the two HPCP vectors;
normalizing the HPCP vectors; and
adding the two HPCP vectors together and normalizing the resultant HPCP vector.

### Claim 9 (depends on 8)

The method of claim 8, wherein filtering one or more bands of spectral peaks includes performing band preset and frequency filtering for a high frequency band of peaks and a low frequency band of peaks.

### Claim 10 (depends on 7)

The method of claim 7, wherein, in extracting a chord detection descriptor, a key detection descriptor, and a local tonality detection descriptor for each audio piece, the method includes:
obtaining an HPCP vector;
averaging the HPCP vector over a time period;
extracting a chord corresponding to the averaged HPCP vector by correlating the averaged HPCP vector with a set of tonic triad tonal profiles; and
repeating the obtaining, averaging and extracting steps for successive audio frames in the audio piece to obtain a sequence of chords or an estimated key for the audio piece.

### Claim 11 (depends on 7)

The method of claim 7, wherein, in extracting a cover version descriptor from each audio piece, the method includes:
obtaining an HPCP vector over a number of window frames;
normalizing the HPCP vector and obtaining a sequence of HPCP vectors;
storing the sequence of HPCP vectors in an HPCP matrix;
calculating a mean value of the HPCP vectors over all frames or by averaging consecutive HPCP vectors;
calculating a transposition index of two songs;
creating a similarity index and determining a measure of similarity;
calculating a similarity matrix; and
obtaining a local alignment matrix from the similarity matrix.

### Claim 12 (depends on 7)

The method of claim 7, wherein, in extracting a western music descriptor from each audio piece, the method includes:
obtaining an HPCP vector over a number of window frames;
obtaining a reference frequency by analyzing the deviation of spectral peaks to a standard frequency;
obtaining a global value by combining frame estimates in a histogram;
extracting a set of local maxima;
calculating an equal tempered deviation;
performing a spectral analysis; and
calculating the dissonance of the audio piece.

### Claim 13 (depends on 2)

The method of claim 2, wherein the rhythm descriptors include an onset rate descriptor, a beats per minute descriptor, a beats loudness descriptor, a bass beats loudness descriptor, and/or a rhythmic intensity descriptor.

### Claim 14 (depends on 13)

The method of claim 13, wherein, in extracting an onset rate descriptor from each audio piece, the method includes:
dividing a digitized audio piece into overlapping window frames;
eliminating or reducing noisy frequencies within the window frames;
performing a frequency quantization of each window frame and acquiring a vector;
calculating an onset detection function including a high frequency content function and a complex domain function;
normalizing the high frequency content function and complex domain function;
comparing each onset detection function value to a dynamic threshold;
calculating a threshold and defining a binary function;
cleaning the results of the binary function; and
calculating an onset rate for the audio piece.

### Claim 15 (depends on 13)

The method of claim 13, wherein, in extracting a beats per minute descriptor from each audio piece, the method includes:
dividing a digitized audio piece into overlapping window frames;
eliminating or reducing noisy frequencies within the window frames;
performing a frequency quantization of each window frame and acquiring a vector;
dividing the spectrum into different bands and computing the energy of each band;
calculating an onset detection function including a high frequency content function and a complex domain function;
resampling the band energy derivatives and onset detection functions;
calculating a temporal unbiased autocorrelation function;
estimating the tempo of the audio piece by selecting the lag of a particular peak;
calculating the beats position by determining phase;
obtaining a tempo period for each of the window frames;
obtaining a sequence of tempi and a sequence of phases; and
obtaining a beats per minute value from the tempo period.

### Claim 16 (depends on 13)

The method of claim 13, wherein, in extracting a beats loudness descriptor from each audio piece, the method includes:
determining a beat attack position;
obtaining an audio frame starting from the beat attack position;
eliminating or reducing noisy frequencies within the audio frame;
calculating a total energy of the beat;
repeating the determining, obtaining, eliminating, and calculating steps to obtain each beat in the audio piece; and
averaging the energy of the audio frames each corresponding to one beat in the audio piece to obtain a beat loudness of the audio piece.

### Claim 17 (depends on 13)

The method of claim 13, wherein, in extracting a bass beats loudness descriptor from each audio piece, the method includes:
determining a beat attack position;
obtaining an audio frame starting from the beat attack position;
eliminating or reducing noisy frequencies within the frame;
calculating a total energy of the beat;
repeating the determining, obtaining, eliminating, and calculating steps to obtain each beat in the audio piece;
averaging the energy of the frames each corresponding to one beat in the audio piece to obtain a beat loudness; and
calculating a ratio of the energy of the low frequencies in the audio piece to the total energy in the audio piece to obtain a bass beats loudness of the audio piece.

### Claim 18 (depends on 13)

The method of claim 13, wherein, in extracting a rhythmic intensity descriptor from each audio piece, the method includes:
calculating a beats per minute descriptor, an onset rate descriptor, a beats loudness descriptor, and a bass beats loudness descriptor from the audio piece;
splitting each descriptor into three different zones;
calculating the rhythmic intensity by assigning a score depending on which zone the value corresponding to the descriptor falls in; and
calculating the sum of the scores for each descriptor and normalizing those values to obtain the rhythmic intensity of the audio piece.

### Claim 19 (depends on 2)

The method of claim 2, wherein the spatial descriptors includes a panning descriptor.

### Claim 20 (depends on 19)

The method of claim 19, wherein, in extracting a panning descriptor from each audio piece, the method includes:
extracting panning coefficients from the audio piece; and
classifying the audio piece based on the panning coefficients.

### Claim 21 (depends on 20)

The method of claim 20, wherein the panning coefficients are extracted from each audio piece by:
performing a frequency quantization of each of a plurality of audio channels within the audio piece;
determining the spatial location of the frequencies;
computing an energy weight histogram;
averaging the energy weight histogram;
normalizing the averaged energy weight histogram; and
converting the normalized histogram into panning coefficients.

### Claim 22 (depends on 1)

The method of claim 1, further comprising classifying each of the audio pieces based on the one or more extracted descriptors.

### Claim 23

A method of detecting cover versions of a song, comprising:
extracting a Harmonic Pitch Class Profile (HPCP) vector from an audio piece;
normalizing the HPCP vector and obtaining a sequence of HPCP vectors;
calculating the mean value of the HPCP vectors over all frames or by averaging consecutive HPCP vectors;
calculating a transposition index of two songs;
creating a similarity index and determining a measure of similarity between the two songs;
calculating a similarity matrix;
obtaining a local alignment matrix from the similarity matrix; and
outputting a result to a user or process indicating the similarity between the two songs.

### Claim 24

A music processing system, comprising:
an input device for receiving an audio signal containing an audio piece;
a tonality analysis module configured to extract tonal features from the audio signal;
a data storing device adapted to store the extracted tonal features;
a tonality comparison device configured to compare the extracted tonal features from the audio signal to tonal features from one or more reference audio pieces; and
an interface for providing a list of audio pieces to a user or process.

### Claim 25 (depends on 24)

The music processing system of claim 24, wherein the tonality analysis module is configured to extract one or more additional descriptors from the audio signal.

### Claim 26 (depends on 25)

The music processing system of claim 25, wherein the one or more additional descriptors includes a dissonance descriptor.

### Claim 27 (depends on 25)

The music processing system of claim 25, wherein the one or more additional descriptors includes a rhythmic descriptor.

### Claim 28 (depends on 25)

The music processing system of claim 25, wherein the one or more additional descriptors includes a spatial descriptor.

### Claim 29 (depends on 24)

The music processing system of claim 24, wherein the tonality comparison device is configured to determine whether the audio piece is a cover version of at least one of the reference audio pieces.
