# US 2015/0310008 A1 · Clustering and synchronizing multimedia contents

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2015/0310008 A1
- Document ID: US20150310008A1
- Status: Published application
- Kind: A1
- Application number: 14/648,705
- Filed: 2013-10-30
- Published: 2015-10-29
- Earliest claimed: 2012-11-30
- Assignees: Thomason Licensing S.A.
- Applicants: Thomason Licensing
- Inventors: Franck THUDOR; Pierre HELLIER; Alexey OZEROV; Ashish BAGRI
- Web page: https://patentgrep.com/patent/US20150310008A1

## CPC

- G06F 16/433: {using audio data}
- G06F 16/27: Replication, distribution or synchronisation of data between databases or within a distributed database system; Distributed database system architectures therefor
- G06F 16/285: {Clustering or classification}
- G10L 25/24: the extracted parameters being the cepstrum
- G10L 25/51: for comparison or discrimination
- G11B 27/10: Indexing; Addressing; Timing or synchronising; Measuring tape travel
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G11B: INFORMATION STORAGE BASED ON RELATIVE MOVEMENT BETWEEN RECORD CARRIER AND TRANSDUCER (recording measured values in a way that does not require playback through a transducer G01D9/00; recording or playback apparatus using mechanically marked tape, e.g. punched paper tape, or using unit records, e.g. punched or magnetically marked cards G06K; transferring data from one type of record carrier to another G06K1/18; circuits for coupling output of reproducer to radio receiver H04B1/20; gramophone pick-ups or like acoustic electromechanical transducers or circuits therefor H04R)

## Abstract

A method and a device for clustering sequences of multimedia contents with regard to a certain event are recommended wherein mel-frequency cepstrum coefficients of the sequences audio tracks of the multimedia contents are used for clustering and synchronizing multimedia contents with regard to a certain event by computing salient mel-frequency cepstrum coefficients from mel-frequency cepstrum coefficient features and clustering sequences having an overlapping audio segment by comparing the salient mel-frequency cepstrum coefficients. Method and device provide an improvement in comparison to fingerprint detection.

## Claims

### Claim 1

Method for clustering sequences of multimedia contents with regard to a certain multimedia presentation event wherein mel-frequency cepstrum coefficients of audio tracks of the multimedia contents are used for clustering and synchronizing multimedia contents with regard to a certain event by
computing salient mel-frequency cepstrum coefficients from mel-frequency cepstrum coefficient features and clustering sequences having an overlapping audio segment by comparing the salient mel-frequency cepstrum coefficients.

### Claim 2 (depends on 1)

Method according to claim 1, wherein the mel-frequency cepstrum coefficient features are mel-frequency cepstrum coefficient vectors.

### Claim 3 (depends on 1)

Method according to claim 1 further comprising a synchronization by comparing sequences of the same cluster with regard to a time offset.

### Claim 4 (depends on 1)

Method according to claim 1 further comprising a final clustering by categorizing sequences into events as sequences which have an overlapping segment form a part of the same event and sequences which do not overlap but are connected via a common sequence also form part of the same event.

### Claim 5 (depends on 1)

Method according to claim 1, wherein said salient mel-frequency cepstrum coefficients are computed as dimension-wise maxima over a predetermined window from the mel-frequency cepstrum coefficients.

### Claim 6 (depends on 1)

Method according to claim 1, wherein said mel-frequency cepstrum coefficient features are compared with regard to whether a majority of features corresponds to a maximum correlation.

### Claim 7 (depends on 6)

Method according to claim 6, wherein the comparing is a result of a voting approach function of the mel-frequency cepstrum coefficient features.

### Claim 8 (depends on 1)

Method according to claim 1, wherein cluster representatives are generated by matching the longest sequences with others to form intermediate clusters in a salient mel-frequency cepstrum coefficient domain.

### Claim 9 (depends on 8)

Method according to claim 8, wherein a created cluster contains one or more cluster representative and comprises the adding of a new audio or audiovisual segment to the created cluster if a new audio or audiovisual segment matches the one or more representatives.

### Claim 10

Device configured to cluster sequences of multimedia contents with regard to a certain multimedia presentation event comprising:
extracting means configured to extract mel-frequency cepstrum coefficients from the sequences audio tracks of the multimedia contents,
computing means configured to calculate dimension-wise maxima over a predetermined window from the mel-frequency cepstrum coefficients to provide salient mel-frequency cepstrum coefficients,
comparing means configured to compare the features of the salient mel-frequency cepstrum coefficients with regard to that a majority of features correspond to a maximum correlation for creating clusters such that every pair of segments having an overlapping audio segment belong to the same cluster.

### Claim 11 (depends on 10)

Device according to claim 10, comprising voting means configured to determine cluster representatives by matching the longest sequences with others to form intermediate clusters in the salient mel-frequency cepstrum coefficient domain.

### Claim 12 (depends on 10)

Device according to claim 10, further comprising:
synchronizing means configured to pair wise compare between all sequences belonging to the same intermediate cluster to provide a complete match-list with time offset between the matching sequences.

### Claim 13 (depends on 10)

Device according to claim 10, further comprising: sorting means configured to categorize sequences into events for final clustering.

### Claim 14 (depends on 10)

Device according to claim 10 wherein the device configured to clustering sequences of multimedia contents with regard to a certain event is a processor-controlled machine.
