# US 12,217,751 B2 · Digital signal processor-based continued conversation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,217,751 B2 - Document ID: US12217751B2 - Status: Granted - Kind: B2 - Application number: 17/644,394 - Filed: 2021-12-15 - Published: 2023-04-13 - Granted: 2025-02-04 - Earliest claimed: 2021-10-13 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Victor Carbune; Matthew Sharifi - Web page: https://patentgrep.com/patent/US12217751B2 ## CPC - G10L 15/1822: {Parsing for meaning understanding} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 16/90332: {Natural language query formulation or dialogue systems} - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L 15/28: Constructional details of speech recognition systems - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 17/06: Decision making techniques; Pattern matching strategies - G10L 25/78: Detection of presence or absence of voice signals (switching of direction of transmission by voice frequency in two-way loud-speaking telephone systems H04M9/10) - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract A method includes instructing an always-on first processor to operate in a follow-on query detection mode, and while the always-on first processor operates in the follow-on query detection mode: receiving follow-on audio data captured by the assistant-enabled device; determining, using a voice activity detection (VAD) model executing on the always-on first processor, whether or not the VAD model detects voice activity in the follow-on audio data; performing, using a speaker identification (SID) model executing on the always-on first processor, speaker verification on the follow-on audio data to determine whether the follow-on audio data includes an utterance spoken by the same user. The method also includes initiating a wake-up process on a second processor to determine whether the utterance includes a follow-on query. ## Claims ### Claim 1 A computer-implemented method when executed on data processing hardware of an assistant-enabled device causes the data processing hardware to perform operations comprising: in response to receiving a response to an initial query submitted by a user of the assistant-enabled device to a digital assistant, instructing an always-on first processor of the data processing hardware to operate in a follow-on query detection mode and an active second processor of the data processing hardware to return to a sleep state; and while the always-on first processor operates in the follow-on query detection mode: receiving, at the always-on first processor, follow-on audio data captured by the assistant-enabled device in an environment of the assistant-enabled device, the follow-on audio data not containing a hotword; determining, using a voice activity detection (VAD) model executing on the always-on first processor, whether or not the VAD model detects voice activity in the follow-on audio data; performing, using a speaker identification (SID) model executing on the always-on first processor, speaker verification on the follow-on audio data to determine whether the follow-on audio data comprises an utterance spoken by the same user that submitted the initial query to the digital assistant; and when the VAD model detects voice activity in the follow-on audio data and the follow-on audio data comprises the utterance spoken by the same user that submitted the initial query, initiating a wake-up process on a second processor of the data processing hardware to determine whether the utterance comprises a follow-on query directed toward the digital assistant. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein instructing the always-on first processor of the data processing hardware to operate in the follow-on query detection mode causes the always-on first processor to initiate execution of the VAD and SID models on the always-on first processor during operation in the follow-on query detection mode. ### Claim 3 (depends on 1) The computer-implemented method of claim 1, wherein instructing the always-on first processor of the data processing hardware to operate in the follow-on query detection mode causes the always-on first processor to disable a hotword detection model during operation in the follow-on query detection mode. ### Claim 4 (depends on 1) The computer-implemented method of claim 1, wherein initiating the wake-up process on the second processor causes the second processor to perform operations comprising: processing the follow-on audio data to generate a transcription of the utterance spoken by the same user that submitted the initial query; and performing query interpretation on the transcription to determine whether or not the utterance comprises the follow-on query directed toward the digital assistant. ### Claim 5 (depends on 4) The computer-implemented method of claim 4, wherein the operations further comprise, when the utterance comprises the follow-on query directed toward the digital assistant: instructing the digital assistant to perform an operation specified by the follow-on query; receiving, from the digital assistant, a follow-on response indicating performance of the operation specified by the follow-on query; and presenting, for output from the assistant-enabled device, the follow-on response. ### Claim 6 (depends on 1) The computer-implemented method of claim 1, wherein initiating the wake-up process on the second processor causes the second processor to transmit the follow-on audio data to a remote server via a network, the follow-on audio data when received by the remote server causing the remote server to perform operations comprising: processing the follow-on audio data to generate a transcription of the utterance spoken by the same user that submitted the initial query; performing query interpretation on the transcription to determine whether or not the utterance comprises the follow-on query directed toward the digital assistant; and when the utterance comprises the follow-on query directed toward the digital assistant, instructing the digital assistant to perform an operation specified by the follow-on query. ### Claim 7 (depends on 6) The computer-implemented method of claim 6, wherein the operations further comprise, after instructing the digital assistant to perform the operation specified by the follow-on query: receiving, from the digital assistant, a follow-on response indicating performance of the operation specified by the follow-on query; and presenting, for output from the assistant-enabled device, the follow-on response. ### Claim 8 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise, after receiving initial audio data corresponding to the initial query spoken by the user and submitted to the digital assistant: extracting, from the initial audio data, a first speaker-discriminative vector representing characteristics of the initial query spoken by the user, wherein performing speaker verification on the follow-on audio data comprises: extracting, from the follow-on audio data, using the SID model, a second speaker-discriminative vector representing characteristics of the follow-on audio data; and determining that the follow-on audio data comprises the utterance spoken by the same user that submitted the initial query to the digital assistant when the first speaker-discriminative vector matches the second speaker-discriminative vector. ### Claim 9 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise, after receiving initial audio data corresponding to the initial query spoken by the user and submitted to the digital assistant: extracting, from the initial audio data, a first speaker-discriminative vector representing characteristics of the initial query spoken by the user; determining whether the first speaker-discriminative vector matches any enrolled speaker vectors stored on the assistant-enabled device, each enrolled speaker vector associated with a different respective enrolled user of the assistant-enabled device; and when the first speaker-discriminative vector matches one of the enrolled speaker vectors, identifying the user that spoke the initial query as the respective enrolled user associated with the one of the enrolled speaker vectors that matches the first speaker-discriminative vector, wherein performing speaker verification on the follow-on audio data comprises: extracting, from the follow-on audio data, using the SID model, a second speaker-discriminative vector representing characteristics of the follow-on audio data; and determining that the follow-on audio data comprises the utterance spoken by the same user that submitted the initial query to the digital assistant when the second speaker-discriminative vector matches the one of the enrolled speaker vectors associated with the respective enrolled user that spoke the initial query. ### Claim 10 (depends on 1) The computer-implemented method of claim 1, wherein the SID model comprises a text-independent SID model configured to extract a text-independent speaker-discriminative vector from the follow-on audio data. ### Claim 11 (depends on 1) The computer-implemented method of claim 1, wherein the operations further comprise, in response to determining at least one of the VAD model failing to detect voice activity in the follow-on audio data or the follow-on audio data fails to comprise an utterance spoken by the same user that submitted the initial query, instructing the always-on first processor to cease operating in the follow-on query detection mode and commence operating in a hotword detection mode. ### Claim 12 (depends on 11) The computer-implemented method of claim 11, wherein instructing the always-on first processor to cease operating in the follow-on query detection mode causes the always-on first processor to disable or deactivate execution of the VAD and SID models on the always-on first processor. ### Claim 13 (depends on 11) The computer-implemented method of claim 11, wherein instructing the always-on first processor to commence operating in the hotword detection mode causes the always-on first processor to initiate execution of a hotword detection model on the always-on first processor. ### Claim 14 (depends on 1) The computer-implemented method of claim 1, wherein: the always-on first processor comprises a digital signal processor (DSP); and the second processor comprises an application processor. ### Claim 15 (depends on 1) The computer-implemented method of claim 1, wherein the assistant-enabled device comprises a battery-powered device in communication with one or more microphones configured to capture the follow-on audio data and initial audio data corresponding to the initial query. ### Claim 16 An assistant-enabled device (AED) comprising: data processing hardware; and memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising: in response to receiving a response to an initial query submitted by a user of the assistant-enabled device to a digital assistant, instructing an always-on first processor of the data processing hardware to operate in a follow-on query detection mode and an active second processor of the data processing hardware to return to a sleep state; and while the always-on first processor operates in the follow-on query detection mode: receiving, at the always-on first processor, follow-on audio data captured by the assistant-enabled device in an environment of the assistant-enabled device, the follow-on audio data not containing a hotword; determining, using a voice activity detection (VAD) model executing on the always-on first processor, whether or not the VAD model detects voice activity in the follow-on audio data; performing, using a speaker identification (SID) model executing on the always-on first processor, speaker verification on the follow-on audio data to determine whether the follow-on audio data comprises an utterance spoken by the same user that submitted the initial query to the digital assistant; and when the VAD model detects voice activity in the follow-on audio data and the follow-on audio data comprises the utterance spoken by the same user that submitted the initial query, initiating a wake-up process on a second processor of the data processing hardware to determine whether the utterance comprises a follow-on query directed toward the digital assistant. ### Claim 17 (depends on 16) The AED of claim 16, wherein instructing the always-on first processor of the data processing hardware to operate in the follow-on query detection mode causes the always-on first processor to initiate execution of the VAD and SID models on the always-on first processor during operation in the follow-on query detection mode. ### Claim 18 (depends on 16) The AED of claim 16, wherein instructing the always-on first processor of the data processing hardware to operate in the follow-on query detection mode causes the always-on first processor to disable a hotword detection model during operation in the follow-on query detection mode. ### Claim 19 (depends on 16) The AED of claim 16, wherein initiating the wake-up process on the second processor causes the second processor to perform operations comprising: processing the follow-on audio data to generate a transcription of the utterance spoken by the same user that submitted the initial query; and performing query interpretation on the transcription to determine whether or not the utterance comprises the follow-on query directed toward the digital assistant. ### Claim 20 (depends on 19) The AED of claim 19, wherein the operations further comprise, when the utterance comprises the follow-on query directed toward the digital assistant: instructing the digital assistant to perform an operation specified by the follow-on query; receiving, from the digital assistant, a follow-on response indicating performance of the operation specified by the follow-on query; and presenting, for output from the assistant-enabled device, the follow-on response. ### Claim 21 (depends on 16) The AED of claim 16, wherein initiating the wake-up process on the second processor causes the second processor to transmit the follow-on audio data to a remote server via a network, the follow-on audio data when received by the remote server causing the remote server to perform operations comprising: processing the follow-on audio data to generate a transcription of the utterance spoken by the same user that submitted the initial query; performing query interpretation on the transcription to determine whether or not the utterance comprises the follow-on query directed toward the digital assistant; and when the utterance comprises the follow-on query directed toward the digital assistant, instructing the digital assistant to perform an operation specified by the follow-on query. ### Claim 22 (depends on 21) The AED of claim 21, wherein the operations further comprise, after instructing the digital assistant to perform the operation specified by the follow-on query: receiving, from the digital assistant, a follow-on response indicating performance of the operation specified by the follow-on query; and presenting, for output from the assistant-enabled device, the follow-on response. ### Claim 23 (depends on 16) The AED of claim 16, wherein the operations further comprise, after receiving initial audio data corresponding to the initial query spoken by the user and submitted to the digital assistant: extracting, from the initial audio data, a first speaker-discriminative vector representing characteristics of the initial query spoken by the user, wherein performing speaker verification on the follow-on audio data comprises: extracting, from the follow-on audio data, using the SID model, a second speaker-discriminative vector representing characteristics of the follow-on audio data; and determining that the follow-on audio data comprises the utterance spoken by the same user that submitted the initial query to the digital assistant when the first speaker-discriminative vector matches the second speaker-discriminative vector. ### Claim 24 (depends on 16) The AED of claim 16, wherein the operations further comprise, after receiving initial audio data corresponding to the initial query spoken by the user and submitted to the digital assistant: extracting, from the initial audio data, a first speaker-discriminative vector representing characteristics of the initial query spoken by the user; determining whether the first speaker-discriminative vector matches any enrolled speaker vectors stored on the assistant-enabled device, each enrolled speaker vector associated with a different respective enrolled user of the assistant-enabled device; and when the first speaker-discriminative vector matches one of the enrolled speaker vectors, identifying the user that spoke the initial query as the respective enrolled user associated with the one of the enrolled speaker vectors that matches the first speaker-discriminative vector, wherein performing speaker verification on the follow-on audio data comprises: extracting, from the follow-on audio data, using the SID model, a second speaker-discriminative vector representing characteristics of the follow-on audio data; and determining that the follow-on audio data comprises the utterance spoken by the same user that submitted the initial query to the digital assistant when the second speaker-discriminative vector matches the one of the enrolled speaker vectors associated with the respective enrolled user that spoke the initial query. ### Claim 25 (depends on 16) The AED of claim 16, wherein the SID model comprises a text-independent SID model configured to extract a text-independent speaker-discriminative vector from the follow-on audio data. ### Claim 26 (depends on 16) The AED of claim 16, wherein the operations further comprise, in response to determining at least one of the VAD model failing to detect voice activity in the follow-on audio data or the follow-on audio data fails to comprise an utterance spoken by the same user that submitted the initial query, instructing the always-on first processor to cease operating in the follow-on query detection mode and commence operating in a hotword detection mode. ### Claim 27 (depends on 26) The AED of claim 26, wherein instructing the always-on first processor to cease operating in the follow-on query detection mode causes the always-on first processor to disable or deactivate execution of the VAD and SID models on the always-on first processor. ### Claim 28 (depends on 26) The AED of claim 26, wherein instructing the always-on first processor to commence operating in the hotword detection mode causes the always-on first processor to initiate execution of a hotword detection model on the always-on first processor. ### Claim 29 (depends on 16) The AED of claim 16, wherein: the always-on first processor comprises a digital signal processor (DSP); and the second processor comprises an application processor. ### Claim 30 (depends on 16) The AED of claim 16, wherein the assistant-enabled device comprises a battery-powered device in communication with one or more microphones configured to capture the follow-on audio data and initial audio data corresponding to the initial query.