# US 2013/0160038 A1 · Audio Fingerprint for Content Identification

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2013/0160038 A1
- Document ID: US20130160038A1
- Status: Published application
- Kind: A1
- Application number: 13/332,331
- Filed: 2011-12-20
- Published: 2013-06-20
- Earliest claimed: 2011-12-20
- Assignees: Yahoo! Inc.
- Inventors: Malcolm Slaney; Andres Hernandez Schafhauser
- Web page: https://patentgrep.com/patent/US20130160038A1

## CPC

- H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- G06F 16/7834: {using audio features}
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/8106: {involving special audio data, e.g. different tracks for different languages}
- H04N 21/8352: involving content or source identification data, e.g. Unique Material Identifier [UMID]
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Methods and system for identifying multimedia content streaming through a television includes retrieving an audio signal from a multimedia content selected for rendering at the television. The retrieved audio signal is partitioned into a plurality of segments of small intervals. A particular segment is analyzed to identify acoustic modulation and to generate a distinct vector for the particular segment based on the acoustic modulation, wherein the vector defines a unique fingerprint of the particular segment of the audio signal. A content database on a server is queried using the vector of the particular segment to obtain content information for multimedia content that matches the fingerprint of the particular segment. The content information is used to identify the multimedia content and the source of the multimedia content that matches the audio signal received for rendering.

## Claims

### Claim 1

A method for identifying multimedia content streaming through a television, the method executed by a processor of the television, comprising:
retrieving an audio signal from a multimedia content selected for rendering at the television;
partitioning the audio signal into a plurality of segments of small intervals;
analyzing a particular segment to identify acoustic modulations in the particular segment, the analysis generating a distinct vector for the particular segment based on the acoustic modulation, the vector defining an unique audio fingerprint of the particular segment of the audio signal; and
querying a content database on a server using the vector of the particular segment of audio signal to obtain content information for multimedia content that matches the fingerprint of the particular segment, the content information used to obtain information related to the multimedia content from a content provider that matches the audio signal received for rendering.

### Claim 2 (depends on 1)

The method of claim 1, wherein the audio signal is captured from multimedia content streamed to the television by a content provider or from a digital multimedia recording device.

### Claim 3 (depends on 1)

The method of claim 1, wherein the small interval is a predefined interval of about 5 seconds.

### Claim 4 (depends on 1)

The method of claim 1, wherein analyzing further includes,
generating an acoustic spectrogram to identify acoustic modulation characteristics for the particular segment of audio signal at one or more frequencies, wherein the acoustic modulation characteristics are spread over a plurality of channels;
examining the acoustic modulation at each channel to measure magnitude, the magnitude identifying amount of energy in each channel at each frequency; and
computing the vector of the particular segment of audio signal as a function of the measured magnitudes in each channel for each frequency for a time period associated with the particular segment of the audio signal, wherein the vector identifies a plurality of floating point numbers of data points representing the unique fingerprint for the particular segment of audio signal.

### Claim 5 (depends on 4)

The method of claim 4, wherein the examination of the acoustic modulation and measuring of magnitude is done using Fast Fourier Transformation technology.

### Claim 6 (depends on 4)

The method of claim 4, wherein querying further includes,
searching the content database to identify one or more multimedia content with audio segments having data points that are closest to the plurality of floating point numbers of the particular segment of the audio signal, the content database being a repository of pre-computed data points for a plurality of audio segments representing different portions of a plurality of audio signals for multimedia content obtained from a plurality of content providers;
computing distance between data points of each audio segment of the identified multimedia content and the floating point numbers of the particular segment using iterative computation; and
selecting the multimedia content that has data points closest to the floating point numbers, wherein the multimedia content is referenced using a unique identifier.

### Claim 7 (depends on 6)

The method of claim 6, further includes retrieving multimedia content related to the entry from the content provider using the unique identifier, the multimedia content including multimedia content matching the particular segment and additional multimedia content for pre-defined amount of time related to the audio signal currently being rendered at the television, the retrieved multimedia content stored in a local cache of the television for subsequent verification of the audio signal for the content that continues to stream through the television.

### Claim 8 (depends on 6)

The method of claim 6, further includes,
when more than one multimedia content has data points closest to the floating point numbers of the particular segment,
performing additional matching by selecting one or more additional segments of the audio signal for the content currently selected for rendering at the television.

### Claim 9 (depends on 1)

The method of claim 1, further includes,
identifying an event or promotional media related to the multimedia content that is scheduled for rendering from a service database, the event or promotional media identified using information from the fingerprint of the particular segment;
retrieving metadata and assets related to the identified event or promotional media from an ad campaign database; and
assembling an application or multimedia content associated with the event or the promotional media using the retrieved metadata and assets, the assembled application or multimedia content related to event or promotional media rendered alongside the multimedia content related to the audio signal at the television.

### Claim 10

A method for identifying content streaming through a television, the method executed by a processor of the television, comprising:
retrieving an audio signal from a content selected for rendering at the television;
partitioning the audio signal into a plurality of segments of small intervals;
analyzing a particular segment to identify acoustic modulations in the particular segment, the analysis generating a vector for the particular segment based on the acoustic modulation, the vector identifying a plurality of floating point numbers related to data points of the particular segment, the vector defining an unique audio fingerprint of the particular segment of the audio signal;
searching a content database to identify one or more content with audio segments having data points that are closest to the plurality of floating point numbers of the particular segment, the content database being a repository of pre-computed data points for a plurality of audio segments representing different portions of a plurality of audio signals for a plurality of content obtained from a plurality of content providers;
obtaining a content identifier of a content having an audio segment that has data points closest to the floating point numbers of the particular segment;
querying a content provider database using the content identifier for information related to the content with an audio segment that matches the particular audio segment; and
receiving a portion of the content from the content provider database in response to the query, the portion of the content includes content recording matching the particular segment and additional recording for a pre-defined amount of time, the additional recording defining a sequence of audio fingerprints for the multimedia content, the portion of the content and additional recording received from the content provider database used in further matching subsequent segments of the audio signal.

### Claim 11 (depends on 10)

The method of claim 10, wherein analyzing further includes,
generating an acoustic spectrogram to identify acoustic modulation characteristics for the particular segment of audio signal at one or more frequencies, wherein the acoustic modulation characteristics are spread over a plurality of channels;
examining the acoustic modulation at each channel to measure magnitude, the magnitude identifying amount of energy in each channel at each frequency, the examining identifying data points related to the acoustic modulation of the particular segment of audio signal; and
computing the vector of the particular segment of audio signal as a function of the measured magnitudes in each channel for each frequency for a time period associated with the particular segment of the audio signal, wherein the vector identifies a plurality of floating point numbers related to data points of the particular segment, the vector representing the unique fingerprint for the particular segment of audio signal.

### Claim 12 (depends on 10)

The method of claim 10, wherein identifying the content identifier further includes,
computing distance between data points of each content in the content database and corresponding floating point numbers of the audio segment using iterative computation; and
identifying the content with a set of data points that are closest to the corresponding floating point numbers of the audio segment.

### Claim 13 (depends on 10)

The method of claim 10, further includes
storing the portion of the content and additional recording received from the content provider database in a local cache accessible to the processor of the television for further verification of the content of the audio signal streaming through the television.

### Claim 14 (depends on 13)

The method of claim 13, further includes,
generating additional fingerprints for additional segment of the streaming audio signal periodically; and
comparing the additional fingerprints against the fingerprint and the sequence of fingerprint of the content and additional recording stored in the local cache to determine if the streaming audio signals continue to match the content in the local cache.

### Claim 15 (depends on 14)

The method of claim 14, further includes,
when the additional fingerprint does not match the fingerprint of the content stored in the local cache,
clearing the content from the local cache;
initiating a search by querying the content database to identify content that matches the additional segment using the additional fingerprints; and
retrieving content from the content provider database for storing in the local cache for subsequent verification.

### Claim 16 (depends on 10)

The method of claim 10, further includes,
identifying a promotional media related to the content from a service database, the promotional media identified using information from the fingerprint of the particular segment;
retrieving metadata and assets related to the identified promotional media from an ad campaign database; and
assembling multimedia content for the promotional media using the retrieved metadata and assets, the assembled multimedia content related to the promotional media rendered alongside the content related to the audio signal at the television.

### Claim 17

A method for identifying content streaming through a television, the method executed by a processor of the television, comprising:
retrieving a set of audio fingerprints associated with a plurality of contents that are scheduled for rendering;
storing the set of audio fingerprints in a local cache associated with the television;
receiving a request for rendering a content on the television;
retrieving an audio signal for the content selected for rendering at the television;
analyzing a particular segment of the audio signal to identify acoustic modulations in the particular segment, the analysis generating a vector for the particular segment based on the acoustic modulation, the vector identifying a plurality of floating point numbers related to data points of the particular segment, the vector defining an unique audio fingerprint of the particular segment of the audio signal;
determining if a match is found for the audio fingerprint of the particular segment of the audio signal within the local cache by comparing audio fingerprint of the particular segment against the audio fingerprint of the plurality of contents;
when a match is found in the local cache, querying a content provider database using a content identifier of the particular content matching the audio fingerprint of the particular segment to obtain a portion of the particular content; and
rendering the particular content obtained from the content provider database in response to the request from the user.

### Claim 18 (depends on 17)

The method of claim 17, further includes,
when the audio fingerprint of the particular segment of the audio signal does not match the fingerprints of any of the plurality of content stored in the local cache,
forwarding a request to a content database for verification of a potential match of the audio fingerprint associated with the audio signal, wherein the request includes a content identifier for a content from a prior query;
receiving confirmation from the content database of the potential match of the audio fingerprint of the audio signal.

### Claim 19 (depends on 17)

The method of claim 17, further includes,
generating additional fingerprints for additional segment of the streaming audio signal periodically; and
verifying the additional fingerprints continue to match the particular content in the local cache by comparing the additional fingerprints against the corresponding fingerprint of the particular content stored in the local cache.

### Claim 20 (depends on 17)

The method of claim 17, wherein the set of audio fingerprints scheduled for rendering are retrieved periodically and stored in the local cache, and wherein the local cache is cleared prior to storing the retrieved audio fingerprints.