# US 10,311,872 B2 · Utterance classifier > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,311,872 B2 - Document ID: US10311872B2 - Status: Granted - Kind: B2 - Application number: 15/659,016 - Filed: 2017-07-25 - Published: 2019-01-31 - Granted: 2019-06-04 - Earliest claimed: 2017-07-25 - Assignees: Google Inc. - Applicants: Google LLC - Inventors: Nathan David Howard; Gabor Simko; Maria Carolina Parada San Martin; Ramkarthik Kalyanasundaram; Guru Prakash Arumugam; Srinivas Vasudevan - Web page: https://patentgrep.com/patent/US10311872B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G10L 15/16: using artificial neural networks - G10L 15/18: using natural language modelling - G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications - G10L 17/00: Speaker identification or verification techniques - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/227: {of the speaker; Human-factor methodology} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Methods, systems, and apparatus, including computer programs encoded on computer storage media for classification using neural networks. One method includes receiving audio data corresponding to an utterance. Obtaining a transcription of the utterance. Generating a representation of the audio data. Generating a representation of the transcription of the utterance. Providing (i) the representation of the audio data and (ii) the representation of the transcription of the utterance to a classifier that, based on a given representation of the audio data and a given representation of the transcription of the utterance, is trained to output an indication of whether the utterance associated with the given representation is likely directed to an automated assistance or is likely not directed to an automated assistant. Receiving, from the classifier, an indication of whether the utterance corresponding to the received audio data is likely directed to the automated assistant or is likely not directed to the automated assistant. Selectively instructing the automated assistant based at least on the indication of whether the utterance corresponding to the received audio data is likely directed to the automated assistant or is likely not directed to the automated assistant. ## Claims ### Claim 1 A computer-implemented method comprising: receiving, by an automated assistant device of a classification system that includes (i) a classifier server, and (ii) the automated assistant device, audio data corresponding to an utterance; obtaining, by the classifier server, a transcription of the utterance; generating, by the classifier server, an acoustic feature vector corresponding to the audio data; generating, by the classifier server, an acoustic feature vector corresponding to the transcription of the utterance; providing, by the classifier server, (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance to a neural-network-based classifier that, based on a given acoustic feature vector corresponding to audio data and a given acoustic feature vector corresponding to a transcription of an utterance, is trained to output an indication of whether a given utterance associated with the given acoustic feature vector corresponding to the audio data is likely directed to an automated assistant or is likely not directed to an automated assistant; determining, by the classifier server, that the classifier has output an indication that the utterance corresponding to the received audio data is likely directed to the automated assistant; instructing, by the classifier server, the automated assistant device to generate a response to the utterance based at least on the indication that the utterance corresponding to the received audio data is likely directed to the automated assistant; generating, by the automated assistant device and for output, the response to the utterance based on the instruction by the classifier server; and providing, by the automated assistant device, the generated response for output. ### Claim 2 (depends on 1) The computer-implemented method of claim 1, wherein providing (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance to the classifier, further comprises: generating an input acoustic feature vector corresponding to a concatenation between (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the utterance; and providing the concatenated input acoustic feature vector to the classifier. ### Claim 3 (depends on 2) The computer-implemented method of claim 2, wherein generating the input acoustic feature vector corresponding to the concatenation between (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance, further comprises: generating one or more numeric vectors converted from one or more words recognized in the transcription of the utterance; and generating a matrix of numeric vectors comprising the one or more numeric vectors to provide to the classifier. ### Claim 4 (depends on 2) The computer-implemented method of claim 2, further comprising: receiving, from a speech recognizer at a word embedding model, recognizable text corresponding to the utterance; generating, at the word embedding model, the transcription of the utterance from the recognizable text; and providing, from the word embedding model, the transcription of the utterance to the classifier. ### Claim 5 (depends on 1) The computer-implemented method of claim 1, wherein the acoustic feature vector corresponding to the audio data comprises acoustic mel-frequency cepstral coefficients (MFCCs) values. ### Claim 6 (depends on 4) The computer-implemented method of claim 4, further comprising: providing, by a max pooling layer of the word embedding model, the transcription of the utterance comprising embedding units; providing, by an acoustic neural network, the acoustic feature vector corresponding to the audio data comprising audio units; concatenating the embedding units and the audio units to a representation unit input; and providing the representation unit input as input to the classifier. ### Claim 7 (depends on 1) The computer-implemented method of claim 1, wherein the classifier server is in communication with the automated assistant device over a network. ### Claim 8 A classification system that includes (i) a classifier server, and (ii) an automated assistant device, the classification system comprising: one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: receiving, by the automated assistant device, audio data corresponding to an utterance; obtaining, by the classifier server, a transcription of the utterance; generating, by the classifier server, an acoustic feature vector corresponding to the audio data; generating, by the classifier server, an acoustic feature vector corresponding to the transcription of the utterance; providing, by the classifier server, (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance to a neural-network-based classifier that, based on a given acoustic feature vector corresponding to audio data and a given acoustic feature vector corresponding to a transcription of an utterance, is trained to output an indication of whether a given utterance associated with the given acoustic feature vector corresponding to the audio data is likely directed to an automated assistant or is likely not directed to an automated assistant; determining, by the classifier server, that the classifier has output an indication that the utterance corresponding to the received audio data is likely directed to the automated assistant; instructing, by the classifier server, the automated assistant device to generate a response to the utterance based at least on the indication that the utterance corresponding to the received audio data is likely directed to the automated assistant; generating, by the automated assistant device and for output, the response to the utterance based on the instruction by the classifier server; and providing, by the automated assistant device, the generated response for output. ### Claim 9 (depends on 8) The system of claim 8, wherein providing (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance to the classifier, further comprises: generating an input acoustic feature vector corresponding to a concatenation between (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the utterance; and providing the concatenated input acoustic feature vector to the classifier. ### Claim 10 (depends on 9) The system of claim 9, wherein generating the input acoustic feature vector corresponding to the concatenation between (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance, further comprises: generating one or more numeric vectors converted from one or more words recognized in the transcription of the utterance; and generating a matrix of numeric vectors comprising the one or more numeric vectors to provide to the classifier. ### Claim 11 (depends on 9) The system of claim 9, further comprising: receiving, from a speech recognizer at a word embedding model, recognizable text corresponding to the utterance; generating, at the word embedding model, the transcription of the utterance from the recognizable text; and providing, from the word embedding model, the transcription of the utterance to the classifier. ### Claim 12 (depends on 11) The system of claim 11, further comprising: providing, by a max pooling layer of the word embedding model, the transcription of the utterance comprising embedding units; providing, by an acoustic neural network, the acoustic feature vector corresponding to the audio data comprising audio units; concatenating the embedding units and the audio units to a representation unit input; and providing the representation unit input as input to the classifier. ### Claim 13 (depends on 8) The system of claim 8, wherein the acoustic feature vector corresponding to the audio data comprises acoustic mel-frequency cepstral coefficients (MFCCs) values. ### Claim 14 (depends on 8) The system of claim 8, wherein the classifier server is in communication with the automated assistant device over a network. ### Claim 15 A non-transitory computer-readable medium storing software comprising instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising: receiving, by an automated assistant device of a classification system that includes (i) a classifier server, and (ii) the automated assistant device, audio data corresponding to an utterance; obtaining, by the classifier server, a transcription of the utterance; generating, by the classifier server, an acoustic feature vector corresponding to of the audio data; generating, by the classifier server, an acoustic feature vector corresponding to the transcription of the utterance; providing, by the classifier server, (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance to a neural-network-based classifier that, based on a given acoustic feature vector corresponding to audio data and a given acoustic feature vector corresponding to a transcription of an utterance, is trained to output an indication of whether a given utterance associated with the given acoustic feature vector corresponding to the audio data is likely directed to an automated assistant or is likely not directed to an automated assistant; determining, by the classifier server, that the classifier has output an indication that the utterance corresponding to the received audio data is likely directed to the automated assistant; instructing, by the classifier server, the automated assistant device to generate a response to the utterance based at least on the indication that the utterance corresponding to the received audio data is likely directed to the automated assistant; generating, by the automated assistant device and for output, the response to the utterance based on the instruction by the classifier server; and providing, by the automated assistant device, the generated response for output. ### Claim 16 (depends on 15) The computer-readable medium of claim 15, wherein providing (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the transcription of the utterance to the classifier, further comprises: generating an input acoustic feature vector corresponding to a concatenation between (i) the acoustic feature vector corresponding to the audio data and (ii) the acoustic feature vector corresponding to the utterance; and providing the concatenated input acoustic feature vector to the classifier.