# US 12,014,727 B2 · Hotwording by degree > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,014,727 B2 - Document ID: US12014727B2 - Status: Granted - Kind: B2 - Application number: 17/305,806 - Filed: 2021-07-14 - Published: 2023-01-19 - Granted: 2024-06-18 - Earliest claimed: 2021-07-14 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Brett Aladdin Barros; James Flynn; Theo Goguely - Web page: https://patentgrep.com/patent/US12014727B2 ## CPC - G10L 15/08: Speech classification or search - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract A method for a soft acceptance of a hotword receives audio data characterizing a soft hotword event detected by a hotword detector in streaming audio captured by a user device. The method also processes the audio data to determine that the audio data corresponds to a query specifying an action to perform on the user device. Without triggering performance of the action on the user device or the other device, the method provides a notification for output from the user device where the notification prompts a user associated with the user device to provide an affirmative input indication in order to trigger performance of the action on the user device or the other device and, when the user fails to provide the affirmative input indication, instructs the user device or the other device to not perform the action specified by the query. ## Claims ### Claim 1 A computer-implemented method when executed by data processing hardware causes the data processing hardware to perform operations comprising: receiving audio data corresponding to an utterance captured in streaming audio by a user device; determining that the received audio data characterizes a soft hotword event detected by a hotword detector, the hotword detector configured to: generate a confidence score indicating a likelihood that a hotword is present in the audio data; and detect the soft hotword event when the confidence score satisfies a first threshold score and dissatisfies a second threshold score; in response to determining that the received audio data characterizes the soft hotword event, processing the audio data to determine that the audio data corresponds to a query specifying an action to perform on the user device; and based on determining that the audio data characterizes the soft hotword event and without triggering performance of the action on the user device or another device: providing a notification for output from the user device, the notification prompting a user associated with the user device to provide an affirmative input indication in order to trigger performance of the action on the user device or the other device; and when the user fails to provide the affirmative input indication, instructing the user device or the other device to not perform the action specified by the query. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein the hotword detector resides on the user device and is configured to detect the soft hotword event when the confidence score is greater than the first threshold score and is less than the second threshold score, the first threshold score less than the second threshold score. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise: determining whether the affirmative input indication is received within a predetermined duration of time since providing the notification for output from the user device; and determining that the user fails to provide the affirmative input indication when the affirmative input indication is not received within the predetermined duration of time. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise: receiving a negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query; and determining that the user fails to provide the affirmative input indication in response to receiving the negative input indication. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the notification provided for output from the user device also prompts the user to provide the negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein providing the notification for output from the user device comprises instructing a graphical user interface executing on the user device to render the notification as a visual notification for display on a screen of the user device. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the operations further comprise instructing the graphical user interface to remove the visual notification displayed on the screen of the user device when the user fails to provide the affirmative input indication. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein providing the notification for output from the user device comprises instructing the user device to output the notification as an audible notification from a speaker of the user device. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise: receiving the affirmative input indication from the user; and in response to receiving the affirmative input indication, triggering performance of the action specified by the query on the user device or the other device. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein receiving the affirmative input indication from the user comprises: receiving a speech input from the user; and determining the speech input includes one or more terms characterizing an intent for the user device to perform the action specified by the query. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, wherein receiving the affirmative input indication from the user comprises receiving a user input indication indicating selection of an affirmative graphical element displayed on a screen of the user device. ### Claim 12 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise, after providing the notification for output from the user device, activating a warm word model configured to detect, without performing speech recognition, a presence of at least one of: one or more spoken terms characterizing the affirmative input indication; or one or more other spoken terms characterizing a negative input indication, the negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query. ### Claim 13 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise, after providing the notification for output from the user device, instructing an automated speech recognizer in communication with the data processing hardware to operate with reduced functionality sufficient for recognizing at least one of: one or more spoken terms characterizing the affirmative input indication; or one or more other spoken terms characterizing a negative input indication, the negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise, when the user fails to provide the affirmative input indication, deleting any storage of the received audio data. ### Claim 15 A system comprising: data processing hardware; and memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising: receiving audio data corresponding to an utterance captured in streaming audio by a user device; determining that the received audio data characterizes a soft hotword event detected by a hotword detector, the hotword detector configured to: generate a confidence score indicating a likelihood that a hotword is present in the audio data; and detect the soft hotword event when the confidence score satisfies a first threshold score and dissatisfies a second threshold score; in response to determining that the received audio data characterizes the soft hotword event, processing the audio data to determine that the audio data corresponds to a query specifying an action to perform on the user device; and based on determining that the audio data characterizes the soft hotword event and without triggering performance of the action on the user device or another device: providing a notification for output from the user device, the notification prompting a user associated with the user device to provide an affirmative input indication in order to trigger performance of the action on the user device or the other device; and when the user fails to provide the affirmative input indication, instructing the user device or the other device to not perform the action specified by the query. ### Claim 16 (depends on 15) The system of claim 15, wherein the hotword detector resides on the user device and is configured to detect the soft hotword event when the confidence score is greater than the first threshold score and is less than the second threshold score, the first threshold score less than the second threshold score. ### Claim 17 (depends on 15) The system of claim 15, wherein the operations further comprise: determining whether the affirmative input indication is received within a predetermined duration of time since providing the notification for output from the user device; and determining that the user fails to provide the affirmative input indication when the affirmative input indication is not received within the predetermined duration of time. ### Claim 18 (depends on 15) The system of claim 15, wherein the operations further comprise: receiving a negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query; and determining that the user fails to provide the affirmative input indication in response to receiving the negative input indication. ### Claim 19 (depends on 18) The system of claim 18, wherein the notification provided for output from the user device also prompts the user to provide the negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query. ### Claim 20 (depends on 15) The system of claim 15, wherein providing the notification for output from the user device comprises instructing a graphical user interface executing on the user device to render the notification as a visual notification for display on a screen of the user device. ### Claim 21 (depends on 20) The system of claim 20, wherein the operations further comprise instructing the graphical user interface to remove the visual notification displayed on the screen of the user device when the user fails to provide the affirmative input indication. ### Claim 22 (depends on 15) The system of claim 15, wherein providing the notification for output from the user device comprises instructing the user device to output the notification as an audible notification from a speaker of the user device. ### Claim 23 (depends on 15) The system of claim 15, wherein the operations further comprise: receiving the affirmative input indication from the user; and in response to receiving the affirmative input indication, triggering performance of the action specified by the query on the user device or the other device. ### Claim 24 (depends on 23) The system of claim 23, wherein receiving the affirmative input indication from the user comprises: receiving a speech input from the user; and determining the speech input includes one or more terms characterizing an intent for the user device to perform the action specified by the query. ### Claim 25 (depends on 23) The system of claim 23, wherein receiving the affirmative input indication from the user comprises receiving a user input indication indicating selection of an affirmative graphical element displayed on a screen of the user device. ### Claim 26 (depends on 15) The system of claim 15, wherein the operations further comprise, after providing the notification for output from the user device, activating a warm word model configured to detect, without performing speech recognition, a presence of at least one of: one or more spoken terms characterizing the affirmative input indication; or one or more other spoken terms characterizing a negative input indication, the negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query. ### Claim 27 (depends on 15) The system of claim 15, wherein the operations further comprise, after providing the notification for output from the user device, instructing an automated speech recognizer in communication with the data processing hardware to operate with reduced functionality sufficient for recognizing at least one of: one or more spoken terms characterizing the affirmative input indication; or one or more other spoken terms characterizing a negative input indication, the negative input indication indicating that the user does not want the user device or the other device to perform the action specified by the query. ### Claim 28 (depends on 15) The system of claim 15, wherein the operations further comprise, when the user fails to provide the affirmative input indication, deleting any storage of the received audio data.