# US 10,311,876 B2 · Server side hotwording > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,876 B2 - Document ID: US10311876B2 - Status: Granted - Kind: B2 - Application number: 15/432,358 - Filed: 2017-02-14 - Published: 2018-08-16 - Granted: 2019-06-04 - Earliest claimed: 2017-02-14 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Alexander H. Gruenstein; Petar Aleksic; Johan Schalkwyk; Pedro J. Moreno Mengibar - Web page: https://patentgrep.com/patent/US10311876B2 ## CPC - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems - G10L 15/183: using context dependencies, e.g. language models - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media, for detecting hotwords using a server. One of the methods includes receiving an audio signal encoding one or more utterances including a first utterance; determining whether at least a portion of the first utterance satisfies a first threshold of being at least a portion of a key phrase; in response to determining that at least the portion of the first utterance satisfies the first threshold of being at least a portion of a key phrase, sending the audio signal to a server system that determines whether the first utterance satisfies a second threshold of being the key phrase, the second threshold being more restrictive than the first threshold; and receiving tagged text data representing the one or more utterances encoded in the audio signal when the server system determines that the first utterance satisfies the second threshold. ## Claims ### Claim 1 A system comprising one or more computers and one or more storage devices on which are stored instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving, from a client device, an audio signal encoding one or more utterances including one or more first utterances for which the client device determined that at least a portion of the one or more first utterances satisfies a first threshold of being at least a portion of a key phrase; determining whether the one or more first utterances satisfy a second threshold of being at least a portion of the key phrase, the second threshold more restrictive than the first threshold; and sending, to the client device, a result of determining whether the one or more first utterances satisfy the second threshold of being the key phrase, comprising: sending, to the client device, data indicating that the key phrase is not likely included in the audio signal in response to determining that the one or more first utterances do not satisfy the second threshold of being the key phrase, or sending, to the client device, data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 2 (depends on 1) The system of claim 1, wherein sending, to the client device, data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase comprises sending, to the client device, tagged text data representing the one or more utterances encoded in the audio signal. ### Claim 3 (depends on 1) The system of claim 1, the operations comprising analyzing the entire audio signal to determine first data for each of the one or more utterances, wherein sending, to the client device, the data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase comprises sending, to the client device, the first data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 4 (depends on 1) The system of claim 1, wherein determining whether the one or more first utterances satisfy the second threshold of being the key phrase comprises determining, using a language model, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 5 (depends on 4) The system of claim 4, the operations comprising customizing the language model for the key phrase prior to determining, using the language model, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 6 (depends on 5) The system of claim 5, the operations comprising receiving text identifying the key phrase, wherein customizing the language model for the key phrase comprises customizing the language model for the key phrase using the text identifying the key phrase. ### Claim 7 (depends on 5) The system of claim 5, the operations comprising: receiving an identifier; and determining, using the identifier, key phrase data for the key phrase, wherein customizing the language model for the key phrase comprises customizing the language model for the key phrase using the key phrase data. ### Claim 8 (depends on 4) The system of claim 4, wherein determining, using the language model, whether the one or more first utterances satisfy the second threshold of being the key phrase comprises determining, using the language model and an acoustic model, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 9 (depends on 8) The system of claim 8, wherein determining, using the language model and the acoustic model, whether the one or more first utterances satisfy the second threshold of being the key phrase comprises: providing data for the one or more first utterances to the language model to cause the language model generate a first output; providing data for the one or more first utterances to the acoustic model to cause the acoustic model to generate a second output; combining the first output and the second output to generate a combined output; and determining, using the combined output, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 10 (depends on 4) The system of claim 4, the operations comprising selecting the language model for a default key phrase. ### Claim 11 (depends on 10) The system of claim 10, the operations comprising determining whether to use the default key phrase. ### Claim 12 A non-transitory computer storage medium encoded with instructions that, when executed by a computer, cause the computer to perform operations comprising: receiving, from a client device, an audio signal encoding one or more utterances including one or more first utterances for which the client device determined that at least a portion of the one or more first utterances satisfies a first threshold of being at least a portion of a key phrase; determining whether the one or more first utterances satisfy a second threshold of being at least a portion of the key phrase, the second threshold more restrictive than the first threshold; and sending, to the client device, a result of determining whether the one or more first utterances satisfy the second threshold of being the key phrase, comprising: sending, to the client device, data indicating that the key phrase is not likely included in the audio signal in response to determining that the one or more first utterances do not satisfy the second threshold of being the key phrase, or sending, to the client device, data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 13 (depends on 12) The medium of claim 12, wherein sending, to the client device, data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase comprises sending, to the client device, tagged text data representing the one or more utterances encoded in the audio signal. ### Claim 14 (depends on 12) The medium of claim 12, the operations comprising analyzing the entire audio signal to determine first data for each of the one or more utterances, wherein sending, to the client device, the data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase comprises sending, to the client device, the first data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 15 (depends on 12) The medium of claim 12, wherein determining whether the one or more first utterances satisfy the second threshold of being the key phrase comprises determining, using a language model, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 16 (depends on 15) The medium of claim 15, the operations comprising customizing the language model for the key phrase prior to determining, using the language model, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 17 (depends on 16) The medium of claim 16, the operations comprising receiving text identifying the key phrase, wherein customizing the language model for the key phrase comprises customizing the language model for the key phrase using the text identifying the key phrase. ### Claim 18 (depends on 16) The medium of claim 16, the operations comprising: receiving an identifier; and determining, using the identifier, key phrase data for the key phrase, wherein customizing the language model for the key phrase comprises customizing the language model for the key phrase using the key phrase data. ### Claim 19 (depends on 15) The medium of claim 15, wherein determining, using the language model, whether the one or more first utterances satisfy the second threshold of being the key phrase comprises determining, using the language model and an acoustic model, whether the one or more first utterances satisfy the second threshold of being the key phrase. ### Claim 20 A computer-implemented method comprising: receiving, from a client device, an audio signal encoding one or more utterances including one or more first utterances for which the client device determined that at least a portion of the one or more first utterances satisfies a first threshold of being at least a portion of a key phrase; determining whether the one or more first utterances satisfy a second threshold of being at least a portion of the key phrase, the second threshold more restrictive than the first threshold; and sending, to the client device, a result of determining whether the one or more first utterances satisfy the second threshold of being the key phrase, comprising: sending, to the client device, data indicating that the key phrase is not likely included in the audio signal in response to determining that the one or more first utterances do not satisfy the second threshold of being the key phrase, or sending, to the client device, data for the audio signal in response to determining that the one or more first utterances satisfy the second threshold of being the key phrase.