# US 12,602,427 B2 · Transforming data from streaming media > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,602,427 B2 - Document ID: US12602427B2 - Status: Granted - Kind: B2 - Application number: 17/918,974 - Filed: 2021-09-24 - Published: 2023-08-03 - Granted: 2026-04-14 - Earliest claimed: 2020-10-30 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Jaakko Henrikki Tiistola - Web page: https://patentgrep.com/patent/US12602427B2 ## CPC - G06F 16/685: {using automatically derived transcript of audio data, e.g. lyrics (speech recognition G10L15/00)} - G06F 16/61: Indexing; Data structures therefor; Storage structures - G06F 40/279: Recognition of textual entities - G06F 40/30: Semantic analysis - G10L 21/00: Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility (G10L19/00 takes precedence) - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 25/54: for retrieval - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for creating augmented content streams by transforming non-textual content into a form that enables a text-based matching system to select non-textual content are described. In some aspects, a method includes obtaining first audio, storing a text transcription of the first audio in a searchable database. Media content that includes second audio is obtained. The second audio is transformed into textual content. A determination is made, based on a search of the searchable database, that the textual content of the second audio matches the text transcription of the first audio. The first audio is inserted into the media content to create an augmented content stream in response to the determination that the textual content of the second audio matches the text transcription of the first audio. ## Claims ### Claim 1 A method, comprising: obtaining first audio; storing a text transcription of the first audio in a searchable database; obtaining media content that includes second audio; transforming the second audio into textual content; identifying, as an exclusion zone for the second audio, a duration of speech comprising one or more spoken words that occur between a first inaudible tone and a second inaudible tone, wherein the first and second inaudible tones signify that the speech that occurs during the exclusion zone is to be omitted from the textual content of the second audio; removing words spoken during the exclusion zone from the textual content of the second audio without removing the duration of speech from the second audio; determining, based on a search of the searchable database, that remaining textual content of the second audio after removing the words spoken during the exclusion zones matches the text transcription of the first audio; and inserting the first audio into a content slot in the media content to create an augmented content stream in response to the determination that the textual content of the second audio matches the text transcription of the first audio, wherein the content slot occurs after the exclusion zone in the media content. ### Claim 2 (depends on 1) The method of claim 1, further comprising: determining a first context of the first audio based on the text transcription of the first audio; and determining a second context of the second audio based on the textual content of the second audio, wherein, determining that the textual content of the second audio matches the text transcription of the first audio comprises determining that the first context matches the second context. ### Claim 3 (depends on 2) The method of claim 2, wherein transforming the second content into textual content comprises: detecting spoken words in the second audio; analyzing one or more audio characteristics of the second audio; adjusting an importance of one or more words from among the spoken words based on the analysis of the one or more audio characteristics; generating the textual content representing the spoken words; and assigning the adjusted importance of the one or more words to the textual content representing the one or more words, including adjusting a weight for each of the one or more words, wherein determining, based on a search of the searchable database, that remaining textual content of the second audio after removing the words spoken during the exclusion zones matches the text transcription of the first audio comprises selecting the text transcription of the audio based on the weights assigned to the one or more words. ### Claim 4 (depends on 3) The method of claim 3, wherein: analyzing the one or more audio characteristics of the second audio comprises detecting an audible indication of emphasis for the one or more words; adjusting the importance of the one or more words comprises increasing the importance of the one or more words based on the audible indication of emphasis; and determining the second context of the second audio comprises determining a topic of the first audio based on the adjusted importance of the one or more words that is assigned to the textual content representing the one or more words. ### Claim 5 (depends on 4) The method of claim 4, wherein: analyzing one or more audio characteristics comprises differentiating between a first portion of the spoken words that are spoken by a host voice in the second audio and a second portion of the spoken words that are spoken by a guest voice in the second audio; and adjusting the importance of the one or more words comprises increasing the importance of the first portion of the spoken words relative to the importance of the second portion of the spoken words. ### Claim 6 (depends on 1) The method of claim 1, further comprising: detecting, as an entity mentioned in the second audio, a product name in the second audio; determining, based on a location of the product name in the second audio, a first specified amount of time before the product name in the second audio and a second specified amount of time after the product name in the second audio; and omitting, from the textual content of the second audio, additional words spoken within a time period that includes the first and second specified amounts of time of before and after the product name in the second audio. ### Claim 7 (depends on 1) The method of claim 1, wherein inserting the first audio into the media content to create an augmented content stream in response to the determination that the textual content of the second audio matches the text transcription of the first audio comprises: assigning respective weights to words of the remaining textual content based on audible and inaudible characteristics of each of the words; identifying multiple candidate audio files including the first audio that match the remaining textual content; for each of the multiple candidate audio files, determining an aggregate weight based on the weight assigned to each word of the remaining textual content that matches a corresponding word of the candidate audio file; and selecting the first audio from the multiple candidate audio files based on the aggregate weight for each of the multiple candidate audio files. ### Claim 8 (depends on 7) The method of claim 7, wherein: the audible characteristics for each word comprise at least one of (i) emphasis detected in the second audio for the word, (ii) a pitch of the word in the second audio, (iii) a speed at which the word is spoken in the second audio, or (iv) whether the word was spoken by a host voice; and the inaudible characteristics for each word comprise whether the word matches a user profile of a user to which the augmented content stream is provided. ### Claim 9 A system comprising: one or more processors; and one or more memory elements including instructions that, when executed, cause the one or more processors to perform operations including: obtaining first audio; storing a text transcription of the first audio in a searchable database; obtaining media content that includes second audio; transforming the second audio into textual content; identifying, as an exclusion zone for the second audio, a duration of speech comprising one or more spoken words that occur between a first inaudible tone and a second inaudible tone, wherein the first and second inaudible tones signify that the speech that occurs during the exclusion zone is to be omitted from the textual content of the second audio; removing words spoken during the exclusion zone from the textual content of the second audio without removing the duration of speech from the second audio; determining, based on a search of the searchable database, that remaining textual content of the second audio after removing the words spoken during the exclusion zones matches the text transcription of the first audio; and inserting the first audio into a content slot in the media content to create an augmented content stream in response to the determination that the textual content of the second audio matches the text transcription of the first audio, wherein the content slot occurs after the exclusion zone in the media content. ### Claim 10 (depends on 9) The system of claim 9, the operations further comprising: determining a first context of the first audio based on the text transcription of the first audio; and determining a second context of the second audio based on the textual content of the second audio, wherein, determining that the textual content of the second audio matches the text transcription of the first audio comprises determining that the first context matches the second context. ### Claim 11 (depends on 10) The system of claim 10, wherein transforming the second content into textual content comprises: detecting spoken words in the second audio; analyzing one or more audio characteristics of the second audio; adjusting an importance of one or more words from among the spoken words based on the analysis of the one or more audio characteristics; generating the textual content representing the spoken words; and assigning the adjusted importance of the one or more words to the textual content representing the one or more words. ### Claim 12 (depends on 11) The system of claim 11, wherein: analyzing the one or more audio characteristics of the second audio comprises detecting an audible indication of emphasis for the one or more words; adjusting the importance of the one or more words comprises increasing the importance of the one or more words based on the audible indication of emphasis; and determining the second context of the second audio comprises determining a topic of the first audio based on the adjusted importance of the one or more words that is assigned to the textual content representing the one or more words. ### Claim 13 (depends on 12) The system of claim 12, wherein: detecting, as an entity mentioned in the second audio, a product name in the second audio; determining, based on a location of the product name in the second audio, a first specified amount of time before the product name in the second audio and a second specified amount of time after the product name in the second audio; and omitting, from the textual content of the second audio, additional words spoken within a time period that includes the first and second specified amounts of time of before and after the product name in the second audio. ### Claim 14 (depends on 9) The system of claim 9, the operations further comprising: detecting, as an entity mentioned in the second audio, a product name in the second audio; and omitting, from the textual content of the second audio, words spoken within a specified amount of time of a location of the product name in the second audio. ### Claim 15 A non-transitory computer storage medium encoded with instructions that when executed by a distributed computing system cause the distributed computing system to perform operations comprising: obtaining first audio; storing a text transcription of the first audio in a searchable database; obtaining media content that includes second audio; transforming the second audio into textual content; identifying, as an exclusion zone for the second audio, a duration of speech comprising one or more spoken words that occur between a first inaudible tone and a second inaudible tone, wherein the first and second inaudible tones signify that the speech that occurs during the exclusion zone is to be omitted from the textual content of the second audio; removing words spoken during the exclusion zone from the textual content of the second audio without removing the duration of speech from the second audio; determining, based on a search of the searchable database, that remaining textual content of the second audio after removing the words spoken during the exclusion zones matches the text transcription of the first audio; and inserting the first audio into a content slot in the media content to create an augmented content stream in response to the determination that the textual content of the second audio matches the text transcription of the first audio, wherein the content slot occurs after the exclusion zone in the media content. ### Claim 16 (depends on 15) The non-transitory computer storage medium of claim 15, the operations further comprising: determining a first context of the first audio based on the text transcription of the first audio; and determining a second context of the second audio based on the textual content of the second audio, wherein, determining that the textual content of the second audio matches the text transcription of the first audio comprises determining that the first context matches the second context. ### Claim 17 (depends on 16) The non-transitory computer storage medium of claim 16, wherein transforming the second content into textual content comprises: detecting spoken words in the second audio; analyzing one or more audio characteristics of the second audio; adjusting an importance of one or more words from among the spoken words based on the analysis of the one or more audio characteristics; generating the textual content representing the spoken words; and assigning the adjusted importance of the one or more words to the textual content representing the one or more words. ### Claim 18 (depends on 17) The non-transitory computer storage medium of claim 17, wherein: analyzing the one or more audio characteristics of the second audio comprises detecting an audible indication of emphasis for the one or more words; adjusting the importance of the one or more words comprises increasing the importance of the one or more words based on the audible indication of emphasis; and determining the second context of the second audio comprises determining a topic of the first audio based on the adjusted importance of the one or more words that is assigned to the textual content representing the one or more words. ### Claim 19 (depends on 18) The non-transitory computer storage medium of claim 18, wherein: analyzing one or more audio characteristics comprises differentiating between a first portion of the spoken words that are spoken by a host voice in the second audio and a second portion of the spoken words that are spoken by a guest voice in the second audio; and adjusting the importance of the one or more words comprises increasing the importance of the first portion of the spoken words relative to the importance of the second portion of the spoken words. ### Claim 20 (depends on 15) The non-transitory computer storage medium of claim 15, the operations further comprising: detecting, as an entity mentioned in the second audio, a product name in the second audio; determining, based on a location of the product name in the second audio, a first specified amount of time before the product name in the second audio and a second specified amount of time after the product name in the second audio; and omitting, from the textual content of the second audio, additional words spoken a time period that includes the first and second specified amounts of time of before and after the product name in the second audio.