# US 2014/0195557 A1 · Presenting Mobile Content Based on Programming Context > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 2014/0195557 A1 - Document ID: US20140195557A1 - Status: Published application - Kind: A1 - Application number: 14/203,229 - Filed: 2014-03-10 - Published: 2014-07-10 - Earliest claimed: 2010-05-19 - Assignees: Google Inc. - Applicants: Google Inc. - Inventors: Ant Oztaskent; Yaroslav Volovich - Web page: https://patentgrep.com/patent/US20140195557A1 ## CPC - G06F 16/48: Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually - G06F 16/43: Querying - G06F 16/433: {using audio data} - G06F 16/7834: {using audio features} - G06F 16/955: using information identifiers, e.g. uniform resource locators [URL] - H04N 21/233: Processing of audio elementary streams {(monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/23418: {involving operations for analysing video streams, e.g. detecting features or characteristics (television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)} - H04N 21/266: Channel or content management, e.g. generation and management of keys and entitlement messages in a conditional access system, merging a VOD unicast channel into a multicast channel - H04N 21/4126: {The peripheral being portable, e.g. PDAs or mobile phones} - H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)} - H04N 21/4826: {using recommendation lists, e.g. of programmes or channels sorted out according to their score} - H04N 21/4828: {for searching programme descriptors (retrieval of video data G06F16/739)} - H04N 21/6581: {Reference data, e.g. a movie identifier for ordering a movie or a product identifier in a home shopping application} - H04N 21/8352: involving content or source identification data, e.g. Unique Material Identifier [UMID] - H04N 21/84: Generation or processing of descriptive data, e.g. content descriptors {(systems specially adapted for using meta-information in broadcast systems H04H60/73)} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION ## Abstract Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for generating search queries in response to obtaining audio samples on a client device. In one aspect, a method includes the actions of i) receiving audio data from a client device, ii) identifying specific content from captured media based on the received audio data, wherein the identified specific content is associated with the received audio data and the captured media includes at least one of audio media or audio-video media, iii) obtaining additional metadata associated with the identified content, iv) generating a search query based at least in part on the obtained additional metadata, and v) returning one or more search results to the client device, the one or more search results responsive to the search query and associated with the received audio data. ## Claims ### Claim 1 A method comprising: receiving audio data from a client device; determining specific content from captured media that matches the received audio data, wherein the captured media is captured from one or more media sources at a backend system and includes at least one of audio media or audio-video media, and wherein determining the specific content that matches the received audio data includes identifying an advertisement included within the captured media and included within the received audio data media programming associated with the received audio data; obtaining additional information associated with the advertisement; generating a search query based at least in part on the obtained additional information; and returning one or more search results to the client device responsive to the search query. ### Claim 2 (depends on 1) The method of claim 1, wherein client device comprises one of one of a mobile phone, a smartphone, a laptop, a network, or a tablet PC. ### Claim 3 (depends on 1) The method of claim 1, wherein the client device includes a set-top box associated with a presentation of audio media or audio-video media. ### Claim 4 (depends on 1) The method of claim 1, wherein the audio data received from the client device includes an audio fingerprint of media recorded at the client device. ### Claim 5 (depends on 1) The method of claim 1, further comprising obtaining an audio fingerprint of the received audio data. ### Claim 6 (depends on 1) The method of claim 1, wherein the captured media includes a collection of captured digital television broadcasts, the method further comprising: monitoring at least one digital television channel; extracting at least an audio stream from each of the at least one digital television channels; generating, for each of the monitored at least one digital television channels, an audio fingerprint from at least a portion of the extracted audio stream; and storing each of the generated audio fingerprints in an audio fingerprint repository, each generated audio fingerprint associated with a unique content identifier. ### Claim 7 (depends on 6) The method of claim 6, further comprising: deriving at least one additional set of information associated with each extracted audio stream; generating at least one set of content-related metadata associated with each extracted audio stream based on the derived at least one additional set of information; associating each of the at least one set of content-related metadata with the unique content identifier associated with the generated audio fingerprint associated with the corresponding extracted audio stream; and storing each of the at least one set of content-related metadata in a metadata repository. ### Claim 8 (depends on 7) The method of claim 7, wherein: identifying specific content from captured media includes: matching an audio fingerprint generated from the audio data received from the client device with at least one audio fingerprint stored in the audio fingerprint repository, and identifying the unique content identifier associated with the at least one audio fingerprint stored in the audio fingerprint repository; and obtaining additional metadata associated with the identified content includes retrieving content-related metadata associated with the identified unique content identifier from the metadata repository. ### Claim 9 (depends on 1) The method of claim 1, wherein the advertisement is identified as an advertisement using a repetition pattern algorithm during analysis of the captured media. ### Claim 10 A system comprising: one or more computers configured to perform operations comprising: receiving audio data from a client device; determining specific content from captured media that matches the received audio data, wherein the captured media is captured from one or more media sources at a backend system and includes at least one of audio media or audio-video media, and wherein determining the specific content that matches the received audio data includes identifying an advertisement included within the captured media and included within the received audio data media programming associated with the received audio data; obtaining additional information associated with the advertisement; generating a search query based at least in part on the obtained additional information; and returning one or more search results to the client device responsive to the search query. ### Claim 11 (depends on 10) The system of claim 10, wherein client device comprises one of one of a mobile phone, a smartphone, a laptop, a network, or a tablet PC. ### Claim 12 (depends on 10) The method of claim 10, wherein the client device includes a set-top box associated with a presentation of audio media or audio-video media. ### Claim 13 (depends on 10) The system of claim 10, wherein the audio data received from the client device includes an audio fingerprint of media recorded at the client device. ### Claim 14 (depends on 10) The system of claim 10, further configured to perform operations comprising obtaining an audio fingerprint of the received audio data. ### Claim 15 (depends on 10) The system of claim 10, wherein the captured media includes a collection of captured digital television broadcasts, the system further configured to perform operations comprising: monitoring at least one digital television channel; extracting at least an audio stream from each of the at least one digital television channels; generating, for each of the monitored at least one digital television channels, an audio fingerprint from at least a portion of the extracted audio stream; and storing each of the generated audio fingerprints in an audio fingerprint repository, each generated audio fingerprint associated with a unique content identifier. ### Claim 16 (depends on 15) The system of claim 15, further configured to perform operations comprising: deriving at least one additional set of information associated with each extracted audio stream; generating at least one set of content-related metadata associated with each extracted audio stream based on the derived at least one additional set of information; associating each of the at least one set of content-related metadata with the unique content identifier associated with the generated audio fingerprint associated with the corresponding extracted audio stream; and storing each of the at least one set of content-related metadata in a metadata repository. ### Claim 17 (depends on 16) The system of claim 16, wherein: identifying specific content from captured media includes: matching an audio fingerprint generated from the audio data received from the client device with at least one audio fingerprint stored in the audio fingerprint repository, and identifying the unique content identifier associated with the at least one audio fingerprint stored in the audio fingerprint repository; and obtaining additional metadata associated with the identified content includes retrieving content-related metadata associated with the identified unique content identifier from the metadata repository. ### Claim 18 (depends on 10) The system of claim 10, wherein the advertisement is identified as an advertisement using a repetition pattern algorithm during analysis of the captured media. ### Claim 19 A computer storage medium encoded with a computer program, the program comprising instructions that when executed by one or more computers cause the one or more computers to perform operations comprising: receiving audio data from a client device; determining specific content from captured media that matches the received audio data, wherein the captured media is captured from one or more media sources at a backend system and includes at least one of audio media or audio-video media, and wherein determining the specific content that matches the received audio data includes identifying an advertisement included within the captured media and included within the received audio data media programming associated with the received audio data; obtaining additional information associated with the advertisement; generating a search query based at least in part on the obtained additional information; and returning one or more search results to the client device responsive to the search query.