# US 12,738,268 B2 · Hot-word free pre-emption of automated assistant response presentation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,738,268 B2 - Document ID: US12738268B2 - Status: Granted - Kind: B2 - Application number: 18/889,063 - Filed: 2024-09-18 - Published: 2025-01-09 - Granted: 2026-09-15 - Earliest claimed: 2020-05-15 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Pu-Sen Chao; Alex Fandrianto - Web page: https://patentgrep.com/patent/US12738268B2 ## CPC - G10L 15/16: using artificial neural networks - G10L 15/222: {Barge in, i.e. overridable guidance for interrupting prompts} - G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU] - G06N 3/0464: Convolutional networks [CNN, ConvNet] - G06N 3/08: Learning methods - G06N 3/09: Supervised learning - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 17/00: Speaker identification or verification techniques - G10L 21/0208: Noise filtering - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2021/02082: {the noise being echo, reverberation of the speech} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS ## Abstract The presentation of an automated assistant response may be selectively pre-empted in response to a hot-word free utterance that is received during the presentation and that is determined to be likely directed to the automated assistant. The determination that the utterance is likely directed to the automated assistant may be performed, for example, using an utterance classification operation that is performed on audio data received during presentation of the response, and based upon such a determination, the response may be pre-empted with another response associated with the later-received utterance. In addition, the duration that is used to determine when a session should be terminated at the conclusion of a conversation between a user and an automated assistant may be dynamically controlled based upon when the presentation of a response has completed. ## Claims ### Claim 1 A computer-implemented method, comprising: receiving first data associated with a first utterance captured by a microphone and directed to an automated assistant device, wherein at least a portion of the first data causes a first utterance fulfillment operation to be initiated to generate a first response for the first utterance captured by the microphone; receiving second data associated with a second, hot-word free utterance captured by the microphone during presentation of the first response to the first utterance captured by the microphone, wherein the presentation of the first response includes playback of a spoken audio response, and the second, hot-word free utterance is captured by the microphone during the playback of the spoken audio response; and using the first and second data, and during the playback of the spoken audio response: determining that the second, hot-word free utterance is likely directed to the automated assistant device; causing a second utterance fulfillment operation to be initiated to generate a second response for the second, hot-word free utterance; and pre-empting the playback of the spoken audio response of the presentation of the first response with a generated presentation of the second response. ### Claim 2 (depends on 1) The method of claim 1, wherein the second data includes audio data, wherein receiving the second data includes monitoring an audio input during the presentation of the first response, and wherein determining that the second, hot-word free utterance is likely directed to the automated assistant device includes initiating an utterance classification operation using the audio data. ### Claim 3 (depends on 2) The method of claim 2, wherein initiating the utterance classification operation includes providing the audio data to a service that includes a neural network-based classifier trained to output an indication of whether a given utterance is likely directed to an automated assistant. ### Claim 4 (depends on 3) The method of claim 3, wherein the service is configured to obtain a transcription of the second, hot-word free utterance, generate a first, acoustic representation associated with the audio data, generate a second, semantic representation associated with the transcription, and provide the first and second representations to the neural network-based classifier to generate the indication. ### Claim 5 (depends on 4) The method of claim 4, wherein the first and second representations respectively include first and second feature vectors, and wherein the service is configured to provide the first and second representations to the neural network based classifier by concatenating the first and second feature vectors. ### Claim 6 (depends on 3) The method of claim 3, wherein the automated assistant device is a client device, and wherein the service is resident on the automated assistant device. ### Claim 7 (depends on 3) The method of claim 3, wherein the automated assistant device is a client device, and wherein the service is remote from and in communication with the automated assistant device. ### Claim 8 (depends on 2) The method of claim 2, further comprising performing acoustic echo cancellation on the audio data to filter at least a portion of the spoken audio response from the audio data. ### Claim 9 (depends on 2) The method of claim 2, further comprising performing speaker identification on the audio data to identify whether the second, hot-word free utterance is associated with a same speaker as the first utterance. ### Claim 10 (depends on 2) The method of claim 2, further comprising, after pre-empting the playback of the spoken audio response: monitoring the audio input during presentation of the second response; dynamically controlling a monitoring duration during presentation of the second response; and automatically terminating an automated assistant session upon completion of the monitoring duration. ### Claim 11 (depends on 10) The method of claim 10, wherein dynamically controlling the monitoring duration includes automatically extending the monitoring duration for a second time period in response to determining after a first time period that the presentation of the second response is not complete. ### Claim 12 (depends on 11) The method of claim 11, wherein automatically extending the monitoring duration for the second time period includes determining the second time period based upon a duration calculated from completion of the presentation of the second response. ### Claim 13 (depends on 1) The method of claim 1, wherein the second, hot-word free utterance is dependent upon the first utterance, the method further comprising propagating an updated client state for the automated assistant device in response to the first utterance prior to completing presentation of the first response such that generation of the second response is based upon the updated client state. ### Claim 14 (depends on 1) The method of claim 1, wherein pre-empting the playback of the spoken audio response with the generated presentation of the second response includes discontinuing the playback of the spoken audio response. ### Claim 15 (depends on 1) The method of claim 1, further comprising continuing the playback of the spoken audio response after pre-empting the playback of the spoken audio response. ### Claim 16 (depends on 1) The method of claim 1, wherein at least one of determining that the second, hot-word free utterance is likely directed to the automated assistant device, causing the second utterance fulfillment operation to be initiated to generate the second response for the second, hot-word free utterance, and pre-empting the playback of the spoken audio response of the presentation of the first response with the generated presentation of the second response is performed using one or more trained machine learning models. ### Claim 17 (depends on 1) The method of claim 1, wherein each of determining that the second, hot-word free utterance is likely directed to the automated assistant device, causing the second utterance fulfillment operation to be initiated to generate the second response for the second, hot-word free utterance, and pre-empting the playback of the spoken audio response of the presentation of the first response with the generated presentation of the second response is performed using one or more trained machine learning models. ### Claim 18 (depends on 1) The method of claim 1, wherein determining that the second, hot-word free utterance is likely directed to the automated assistant device, causing the second utterance fulfillment operation to be initiated to generate the second response for the second, hot-word free utterance, and pre-empting the playback of the spoken audio response of the presentation of the first response with the generated presentation of the second response using the first and second data further uses the first response. ### Claim 19 A system comprising: one or more processors; and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to: receive first data associated with a first utterance captured by a microphone and directed to an automated assistant device, wherein at least a portion of the first data causes a first utterance fulfillment operation to be initiated to generate a first response for the first utterance captured by the microphone; receive second data associated with a second, hot-word free utterance captured by the microphone during presentation of the first response to the first utterance captured by the microphone, wherein the presentation of the first response includes playback of a spoken audio response, and the second, hot-word free utterance is captured by the microphone during the playback of the spoken audio response; and using the first and second data, and during the playback of the spoken audio response: determine that the second, hot-word free utterance is likely directed to the automated assistant device; cause a second utterance fulfillment operation to be initiated to generate a second response for the second, hot-word free utterance; and pre-empt the playback of the spoken audio response of the presentation of the first response with a generated presentation of the second response. ### Claim 20 An automated assistant device, comprising: a microphone; one or more processors; and memory operably coupled with the one or more processors, wherein the memory stores instructions that, in response to execution of the instructions by one or more processors, cause the one or more processors to perform a method that includes: receiving first data associated with a first utterance captured by the microphone and directed to an automated assistant device, wherein at least a portion of the first data causes a first utterance fulfillment operation to be initiated to generate a first response for the first utterance captured by the microphone; receiving second data associated with a second, hot-word free utterance captured by the microphone during presentation of the first response to the first utterance captured by the microphone, wherein the presentation of the first response includes playback of a spoken audio response, and the second, hot-word free utterance is captured by the microphone during the playback of the spoken audio response; and using the first and second data, and during the playback of the spoken audio response: determining that the second, hot-word free utterance is likely directed to the automated assistant device; causing a second utterance fulfillment operation to be initiated to generate a second response for the second, hot-word free utterance; and pre-empting the playback of the spoken audio response of the presentation of the first response with a generated presentation of the second response.