# US 12,217,740 B2 · Detection and/or enrollment of hot commands to trigger responsive action by automated assistant > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,740 B2 - Document ID: US12217740B2 - Status: Granted - Kind: B2 - Application number: 18/581,286 - Filed: 2024-02-19 - Published: 2024-06-13 - Granted: 2025-02-04 - Earliest claimed: 2019-10-15 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Yuan Yuan; Bibo Xu; Tianyu Wang; Anurag Jain - Web page: https://patentgrep.com/patent/US12217740B2 ## CPC - G10L 15/08: Speech classification or search - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 40/279: Recognition of textual entities - G06N 20/00: Machine learning - G06V 40/19: Sensors therefor - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/226: {using non-speech characteristics} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS - G06V: IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING ## Abstract Techniques are described herein for detecting and/or enrolling (or commissioning) new “hot commands” that are useable to cause an automated assistant to perform responsive action(s) without having to be first explicitly invoked. In various implementations, an automated assistant may be transitioned from a limited listening state into a full speech recognition state in response to a trigger event. While in the full speech recognition state, the automated assistant may receive and perform speech recognition processing on a spoken command from a user to generate a textual command. The textual command may be determined to satisfy a frequency threshold in a corpus of textual commands. Consequently, data indicative of the textual command may be enrolled as a hot command. Subsequent utterance of another textual command that is semantically consistent with the textual command may trigger performance of a responsive action by the automated assistant, without requiring explicit invocation. ## Claims ### Claim 1 A method implemented using one or more processors, comprising: operating an automated assistant using one or more of the processors; transitioning the automated assistant from a limited listening state into a full speech recognition state in response to a trigger event; while in the full speech recognition state, receiving, by the automated assistant, a spoken command from a user; performing speech recognition processing on the spoken command to generate a textual command; applying the textual command as input across a machine learning model to generate output, wherein the output is indicative of a probability that the textual command was directed to the automated assistant; determining that the textual command satisfies a frequency threshold in a corpus of textual commands; and based on the output and the determining, enrolling data indicative of the textual command as a hot command, wherein subsequent to the enrolling, utterance of another textual command that is semantically consistent with the textual command triggers performance of a responsive action by the automated assistant, without requiring explicit invocation of the automated assistant. ### Claim 2 (depends on 1) The method of claim 1, wherein the trigger event comprises detection of one or more default hot words in audio data captured by a microphone. ### Claim 3 (depends on 1) The method of claim 1, wherein the corpus of textual commands comprises a corpus of textual commands generated by the user. ### Claim 4 (depends on 1) The method of claim 1, wherein the corpus of textual commands comprises a corpus of textual commands generated by a population of users, including the user. ### Claim 5 (depends on 1) The method of claim 1, further comprising detecting, based on vision data captured by one or more cameras, that the user's gaze is directed at a computing device on which the automated assistant is at least partially implemented, wherein the enrolling is further in response to the detecting. ### Claim 6 (depends on 1) The method of claim 1, wherein the enrolling is further in response to a determination that, during a predetermined time interval after the spoken command, no additional utterances are detected. ### Claim 7 (depends on 1) The method of claim 1, wherein the enrolling is further in response to a determination that, during a predetermined time interval after the spoken command, no responsive action was taken by a person. ### Claim 8 (depends on 1) The method of claim 1, further comprising, in response to the enrolling, caching data indicative of the textual command or the textual command itself in local memory of a computing device that at least partially implements the automated assistant. ### Claim 9 (depends on 8) The method of claim 8, further comprising triggering the textual command subsequent to the enrolling based on the cached data, without the textual command being explicitly invoked, in order to cache information that is responsive to the textual command, wherein subsequent to the triggering, the cached information is output in response to a subsequent invocation of the textual command, in lieu of the textual command being triggered. ### Claim 10 (depends on 8) The method of claim 8, wherein subsequent to the caching, invocation of the textual command causes the data indicative of the textual command to be transmitted directly to a smart appliance over one or both of a local area network (“LAN”) and a personal area network (“PAN”), wherein receipt of the data indicative of the textual command causes the smart appliance to perform a responsive action. ### Claim 11 (depends on 1) The method of claim 1, wherein the data indicative of the textual command comprises a template that includes one or more slots to receive one or more subsequent parameters that are semantically consistent with one or more parameters provided in the spoken command received from the user. ### Claim 12 A system comprising one or more processors and memory storing instructions that, in response to execution of the instructions by the one or more processors, cause the one or more processors to implement an automated assistant, wherein the automated assistant is to: receive audio data captured at one or more microphones, wherein the audio data includes first and second spoken utterances; perform speech recognition processing on the audio data to generate first and second textual snippets corresponding to the first and second spoken utterances; apply the first and second textual snippets as inputs across a machine learning model to generate first and second outputs, wherein the first and second outputs are indicative of first and second probabilities, respectively, that the first and second textual snippets are directed to the automated assistant; in response to a determination, based on the first output that the first textual snippet satisfies one or more hot command criteria, perform a responsive action based on the first textual snippet; in response to a determination, based on the second output, that the second textual snippet fails to satisfy the one or more hot command criteria, discard or disregard the second textual snippet without performing another responsive action. ### Claim 13 (depends on 12) The system of claim 12, wherein the one or more hot command criteria include enrollment of a textual snippet under consideration in a hot command library, wherein the hot command library includes textual snippets that, when spoken, trigger performance of a responsive action by the automated assistant, without requiring detection of one or more default hot words or transition of the automated assistant into an active listening state. ### Claim 14 (depends on 12) The system of claim 12, further comprising instructions to detect, based on vision data captured by one or more cameras, a first gaze of a user while the user provided the first spoken utterance and a second gaze of the user while the user provided the second spoken utterance. ### Claim 15 (depends on 14) The system of claim 14, wherein the one or more hot command criteria include a gaze being directed at a computing device on which the automated assistant is at least partially implemented. ### Claim 16 (depends on 12) The system of claim 12, wherein the one or more hot command criteria include a determination that, during a predetermined time interval after the spoken command, no additional utterances are detected. ### Claim 17 At least one non-transitory computer-readable medium comprising instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: operating an automated assistant using one or more of the processors; transitioning the automated assistant from a limited listening state into a full speech recognition state in response to a trigger event; while in the full speech recognition state, receiving, by the automated assistant, a spoken command from a user; performing speech recognition processing on the spoken command to generate a textual command; apply the textual command as input across a machine learning model to generate output, wherein the output is indicative of a probability that the textual command was directed to the automated assistant; determining that the textual command satisfies a frequency threshold in a corpus of textual commands; and based on the output and the determining, enrolling data indicative of the textual command as a hot command, wherein subsequent to the enrolling, utterance of another textual command that is semantically consistent with the textual command triggers performance of a responsive action by the automated assistant, without requiring explicit invocation of the automated assistant. ### Claim 18 (depends on 17) The at least one non-transitory computer-readable medium of claim 17, wherein the trigger event comprises detection of one or more default hot words in audio data captured by a microphone. ### Claim 19 (depends on 17) The at least one non-transitory computer-readable medium of claim 17, wherein the corpus of textual commands comprises a corpus of textual commands generated by the user. ### Claim 20 (depends on 17) The at least one non-transitory computer-readable medium of claim 17, wherein the corpus of textual commands comprises a corpus of textual commands generated by a population of users, including the user.