# US 2019/0379927 A1 · Media Content Identification on Mobile Devices

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0379927 A1
- Document ID: US20190379927A1
- Status: Published application
- Kind: A1
- Application number: 16/441,949
- Filed: 2019-06-14
- Published: 2019-12-12
- Earliest claimed: 2012-02-21
- Assignees: Gracenote, Inc.
- Applicants: Gracenote, Inc.
- Inventors: Mihailo M. Stojancic; Sunil Suresh Kulkarni; Shashank Merchant; Jose Pio Pereira; Oleksiy Bolgarov
- Web page: https://patentgrep.com/patent/US20190379927A1

## CPC

- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- G10H 1/0008: {Associated control or indicating means}
- H04N 1/32: Circuits or arrangements for control or supervision between transmitter and receiver {or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device (H04N1/38, H04N1/387 take precedence)}
- H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/43072: {of multiple content streams on the same device}
- H04N 21/43079: {of additional data with content streams on multiple devices}
- H04N 21/4332: {by placing content in organized collections, e.g. local EPG data repository (interfaces, Database management systems or updating for information retrieval G06F16/23; details of retrieval of video data and associated meta data in video database G06F16/739)}
- H04N 21/439: Processing of audio elementary streams
- H04N 21/6581: {Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application}
- H04N 21/8352: involving content or source identification data, e.g. Unique Material Identifier [UMID]
- H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)}
- H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments}
- G10H 2210/041: based on mfcc [mel -frequency spectral coefficients]
- G10H 2210/051: for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings
- G10H 2240/141: Library retrieval matching, i.e. any of the steps of matching an inputted segment or phrase with musical database contents, e.g. query by humming, singing or playing; the steps may include, e.g. musical analysis of the input, musical feature extraction, query formulation, or details of the retrieval process
- G10H 2250/235: Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT]
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION
- G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE

## Abstract

A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification.

## Claims

### Claim 1

A computer-implemented method comprising:
determining, by computing device, a match point within a sequence of media content that is presented by a playback device, wherein the match point is associated with a reference fingerprint of a set of reference fingerprints, and wherein the set of reference fingerprints corresponds to the sequence of media content and is stored in a memory of the computing device;
obtaining, by the computing device, a fragment of media content;
generating, by the computing device, a query fingerprint using the fragment of media content;
comparing, by the computing device, the query fingerprint with a first subset of the set of reference fingerprints, wherein each reference fingerprint of the first subset corresponds to a portion of the sequence of media content that is adjacent to the match point;
based on a result of the comparing, determining, by the computing device, that the query fingerprint does not match any reference fingerprints of the first subset; and
based on the determining that the query fingerprint does not match any reference fingerprints of the first subset, comparing, by the computing device, the query fingerprint with a second subset of the set of reference fingerprints, wherein each reference fingerprint of the second subset has a respective timestamp that is within a threshold time span of the match point, and wherein the second subset has more reference fingerprints than the first subset.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining that the query fingerprint matches a given reference fingerprint of the second subset; and
based on the determining that the query fingerprint matches the given reference fingerprint, triggering an action event on the computing device.

### Claim 3 (depends on 2)

The computer-implemented method of claim 2, further comprising obtaining data corresponding to the given reference fingerprint, wherein the action event is based on the data.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, further comprising:
obtaining, by the computing device from a server device, the set of reference fingerprints; and
storing, by the computing device, the set of reference fingerprints in the memory of the computing device.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using a microphone, wherein the fragment of media content comprises an audio fragment, and wherein the query fingerprint comprises an audio fingerprint.

### Claim 6 (depends on 5)

The computer-implemented method of claim 5, further comprising resampling the audio fragment or down-mixing the audio fragment prior to generating the audio fingerprint.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using a camera, wherein the fragment of media content comprises a video fragment, and wherein the query fingerprint comprises a video fingerprint.

### Claim 8 (depends on 7)

The computer-implemented method of claim 7, further comprising selecting a video frame of the video fragment, wherein generating the query fingerprint comprises generating the video fingerprint using the video frame.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein the playback device comprises a television.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the portion of the sequence of media content has a length in time that is proportional to a length in time of the fragment of media content.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, further comprising:
determining that the query fingerprint does not match any reference fingerprints of the second subset; and
based on the determining that the query fingerprint does not match any reference fingerprints of the second subset, comparing, by the computing device, the query fingerprint with other reference fingerprints of the set of reference fingerprints that are separate from the second sub set.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11, further comprising:
determining that the query fingerprint does not match any reference fingerprints of the set of reference fingerprints; and
based on the determining that query fingerprint does not match any reference fingerprints of the set of reference fingerprints, transmitting, to a server device, the query fingerprint for comparison with reference fingerprints of a reference database.

### Claim 13

A computing device comprising:
one or more processors; and
a computer-readable medium having stored therein instructions that are executable by the one or more computers to cause the computing device to perform functions comprising:
determining a match point within a sequence of media content that is presented by a playback device, wherein the match point is associated with a reference fingerprint of a set of reference fingerprints, and wherein the set of reference fingerprints corresponds to the sequence of media content and is stored in a memory of the computing device,
obtaining a fragment of media content,
generating a query fingerprint using the fragment of media content,
comparing the query fingerprint with a first subset of the set of reference fingerprints, wherein each reference fingerprint of the first subset corresponds to a portion of the sequence of media content that is adjacent to the match point,
based on a result of the comparing, determining that the query fingerprint does not match any reference fingerprints of the first subset, and
based on the determining that the query fingerprint does not match any reference fingerprints of the first subset, comparing the query fingerprint with a second subset of the set of reference fingerprints, wherein each reference fingerprint of the second subset has a respective timestamp that is within a threshold time span of the match point, and wherein the second subset has more reference fingerprints than the first subset.

### Claim 14 (depends on 13)

The computing device of claim 13, wherein the functions further comprise:
determining that the query fingerprint matches a given reference fingerprint of the second subset; and
based on the determining that the query fingerprint matches the given reference fingerprint, triggering an action event on the computing device.

### Claim 15 (depends on 13)

The computing device of claim 13, wherein the computing device further comprises a microphone, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using the microphone, wherein the fragment of media content comprises an audio fragment, and wherein the query fingerprint comprises an audio fingerprint.

### Claim 16 (depends on 13)

The computing device of claim 13, wherein the computing device further comprises a camera, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using the camera, wherein the fragment of media content comprises a video fragment, and wherein the query fingerprint comprises a video fingerprint.

### Claim 17 (depends on 13)

The computing device of claim 13, wherein the functions further comprise:
determining that the query fingerprint does not match any reference fingerprints of the second subset; and
based on the determining that the query fingerprint does not match any reference fingerprints of the second subset, comparing the query fingerprint with other reference fingerprints of the set of reference fingerprints that are separate from the second subset.

### Claim 18

A non-transitory computer-readable medium having stored therein instructions that are executable by one or more processors to cause a computing device to perform functions comprising:
determining a match point within a sequence of media content that is presented by a playback device, wherein the match point is associated with a reference fingerprint of a set of reference fingerprints, and wherein the set of reference fingerprints corresponds to the sequence of media content and is stored in a memory of the computing device,
obtaining a fragment of media content,
generating a query fingerprint using the fragment of media content,
comparing the query fingerprint with a first subset of the set of reference fingerprints, wherein each reference fingerprint of the first subset corresponds to a portion of the sequence of media content that is adjacent to the match point,
based on a result of the comparing, determining that the query fingerprint does not match any reference fingerprints of the first subset, and
based on the determining that the query fingerprint does not match any reference fingerprints of the first subset, comparing the query fingerprint with a second subset of the set of reference fingerprints, wherein each reference fingerprint of the second subset has a respective timestamp that is within a threshold time span of the match point, and wherein the second subset has more reference fingerprints than the first subset.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein the functions further comprise:
determining that the query fingerprint matches a given reference fingerprint of the second subset; and
based on the determining that the query fingerprint matches the given reference fingerprint, triggering an action event on the computing device.

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein obtaining the fragment of media content comprises obtaining the fragment of media content using a microphone, wherein the fragment of media content comprises an audio fragment, and wherein the query fingerprint comprises an audio fingerprint.
