# US 11,445,242 B2 · Media content identification on mobile devices > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 11,445,242 B2 - Document ID: US11445242B2 - Status: Granted - Kind: B2 - Application number: 16/441,996 - Filed: 2019-06-14 - Published: 2019-12-12 - Granted: 2022-09-13 - Earliest claimed: 2012-02-21 - Assignees: Roku, Inc. - Applicants: Roku, Inc. - Inventors: Mihailo M. Stojancic; Daniel H. Eakins; Shashank Merchant; Prashant Ramanathan; Jose Pio Pereira - Web page: https://patentgrep.com/patent/US11445242B2 ## CPC - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - G10H 1/0008: {Associated control or indicating means} - H04N 1/32: Circuits or arrangements for control or supervision between transmitter and receiver {or between image input and image output device, e.g. between a still-image camera and its memory or between a still-image camera and a printer device (H04N1/38, H04N1/387 take precedence)} - H04N 21/41407: {embedded in a portable device, e.g. video client on a mobile phone, PDA, laptop (constructional details of equipment or arrangements specially adapted for portable computer application G06F1/1626; arrangements specially adapted for mobile receivers in broadcast systems H04H20/57)} - H04N 21/42203: {sound input device, e.g. microphone} - H04N 21/4223: Cameras (H04N23/00 takes precedence) - H04N 21/43072: {of multiple content streams on the same device} - H04N 21/43079: {of additional data with content streams on multiple devices} - H04N 21/4332: {by placing content in organized collections, e.g. local EPG data repository (interfaces, Database management systems or updating for information retrieval G06F16/23; details of retrieval of video data and associated meta data in video database G06F16/739)} - H04N 21/439: Processing of audio elementary streams - H04N 21/6581: {Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application} - H04N 21/8352: involving content or source identification data, e.g. Unique Material Identifier [UMID] - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - H04N 21/8456: {by decomposing the content in the time domain, e.g. in time segments} - G10H 2210/041: based on mfcc [mel -frequency spectral coefficients] - G10H 2210/051: for extraction or detection of onsets of musical sounds or notes, i.e. note attack timings - G10H 2240/141: Library retrieval matching, i.e. any of the steps of matching an inputted segment or phrase with musical database contents, e.g. query by humming, singing or playing; the steps may include, e.g. musical analysis of the input, musical feature extraction, query formulation, or details of the retrieval process - G10H 2250/235: Fourier transform; Discrete Fourier Transform [DFT]; Fast Fourier Transform [FFT] - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION - G10H: ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE ## Abstract A mobile device responds in real time to media content presented on a media device, such as a television. The mobile device captures temporal fragments of audio-video content on its microphone, camera, or both and generates corresponding audio-video query fingerprints. The query fingerprints are transmitted to a search server located remotely or used with a search function on the mobile device for content search and identification. Audio features are extracted and audio signal global onset detection is used for input audio frame alignment. Additional audio feature signatures are generated from local audio frame onsets, audio frame frequency domain entropy, and maximum change in the spectral coefficients. Video frames are analyzed to find a television screen in the frames, and a detected active television quadrilateral is used to generate video fingerprints to be combined with audio fingerprints for more reliable content identification. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by a client media device from a server device, a location-specific advertisement fingerprint set for a particular day-part, wherein the location-specific advertisement fingerprint set comprises query fingerprints that correspond to respective known advertisements that are shown during the particular day-part and at a location of the client media device; storing, by the client media device, the query fingerprints in a query database; generating, by the client media device, reference fingerprints corresponding to respective portions of a media sequence that is presented by the client media device during the particular day-part and at the location of the client media device; storing, by the client media device, the reference fingerprints in a reference database; identifying, by the client media device, a match between a query fingerprint in the query database and a reference fingerprint in the reference database; and based on the identifying the match, initiating an actionable event on the client media device. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the reference database comprises a revolving buffer of a programmable length, and wherein the computer-implemented method further comprises updating the revolving buffer according to a programmable schedule. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein updating the revolving buffer comprises: removing an oldest reference fingerprint from the revolving buffer; and adding a new reference fingerprint to the revolving buffer. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, further comprising: determining that the reference database is full; and based on the determining that the reference database is full, removing an oldest reference fingerprint from the reference database. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein initiating the actionable event comprises replacing a generic advertisement with a targeted advertisement. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein initiating the actionable event comprises overlaying an advertisement with local advertisement information. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein initiating the actionable event comprises providing a menu of enhancement programming options on the client media device. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein initiating the actionable event comprises sending, to a server device, an indication that the client media device presented a known advertisement with which the query fingerprint corresponds. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the client media device is a network-connected client video device, and wherein the reference fingerprints comprise video fingerprints. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the client media device is a network-connected client audio device, and wherein the reference fingerprints comprise audio fingerprints. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein: the particular day-part is morning, and the computer-implemented method further comprises receiving a second location-specific advertisement fingerprint set comprising fingerprints that correspond to respective known advertisement that are shown during a different day-part than the particular day-part at the location of the client media device. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the query fingerprints only correspond to beginning frames of the respective known advertisements. ### Claim 13 A client media device comprising: one or more processors; and a non-transitory computer-readable medium having stored therein instructions that are executable by the one or more processors to cause the client media device to perform functions comprising: receiving, from a server device, a location-specific advertisement fingerprint set for a particular day-part, wherein the location-specific advertisement fingerprint set comprises query fingerprints that correspond to respective known advertisements that are shown during the particular day-part and at a location of the client media device, storing the query fingerprints in a query database, generating reference fingerprints corresponding to respective portions of a media sequence that is presented by the client media device during the particular day-part and at the location of the client media device, storing the reference fingerprints in a reference database, identifying a match between a query fingerprint in the query database and a reference fingerprint in the reference database, based on the identifying the match, initiating an actionable event on the client media device. ### Claim 14 (depends on 13) The client media device of claim 13, wherein the reference database comprises a revolving buffer of a programmable length, and wherein the functions further comprise updating the revolving buffer according to a programmable schedule. ### Claim 15 (depends on 13) The client media device of claim 13, wherein the functions further comprise: determining that the reference database is full; and based on the determining that the reference database is full, removing an oldest reference fingerprint from the reference database. ### Claim 16 (depends on 13) The client media device of claim 13, wherein initiating the actionable event comprises replacing a generic advertisement with a targeted advertisement. ### Claim 17 (depends on 13) The client media device of claim 13, wherein initiating the actionable event comprises sending, to a server device, an indication that the client media device presented a known advertisement with which the query fingerprint corresponds. ### Claim 18 A non-transitory computer-readable medium having stored therein instructions that are executable by one or more processors to cause a client media device to perform functions comprising: receiving, from a server device, a location-specific advertisement fingerprint set for a particular day-part, wherein the location-specific advertisement fingerprint set comprises query fingerprints that correspond to respective known advertisements that are shown during the particular day-part and at a location of the client media device, storing the query fingerprints in a query database, generating reference fingerprints corresponding to respective portions of a media sequence that is presented by the client media device during the particular day-part and at the location of the client media device, storing the reference fingerprints in a reference database, identifying a match between a query fingerprint in the query database and a reference fingerprint in the reference database, based on the identifying the match, initiating an actionable event on the client media device. ### Claim 19 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein the reference database comprises a revolving buffer of a programmable length, and wherein the functions further comprise updating the revolving buffer according to a programmable schedule. ### Claim 20 (depends on 18) The non-transitory computer-readable medium of claim 18, wherein initiating the actionable event comprises replacing a generic advertisement with a targeted advertisement.