# US 9,436,678 B2 · Architecture for multi-domain natural language processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 9,436,678 B2 - Document ID: US9436678B2 - Status: Granted - Kind: B2 - Application number: 14/754,598 - Filed: 2015-06-29 - Published: 2015-10-22 - Granted: 2016-09-06 - Earliest claimed: 2012-12-19 - Assignees: Amazon Technologies, Inc. - Applicants: Amazon Technologies, Inc. - Inventors: Lambert Mathias; Ying Shi; Imre Attila Kiss; Ryan Paul Thomas; Frederic Johan Georges Deramat - Web page: https://patentgrep.com/patent/US9436678B2 ## CPC - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G06F 40/284: Lexical analysis, e.g. tokenisation or collocates - G06F 40/295: Named entity recognition - G06F 40/35: Discourse or dialogue representation - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/56: Natural language generation - G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination - G10L 15/26: Speech to text systems (G10L15/08 takes precedence) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract Features are disclosed for processing a user utterance with respect to multiple subject matters or domains, and for selecting a likely result from a particular domain with which to respond to the utterance or otherwise take action. A user utterance may be transcribed by an automatic speech recognition (“ASR”) module, and the results may be provided to a multi-domain natural language understanding (“NLU”) engine. The multi-domain NLU engine may process the transcription(s) in multiple individual domains rather than in a single domain. In some cases, the transcription(s) may be processed in multiple individual domains in parallel or substantially simultaneously. In addition, hints may be generated based on previous user interactions and other data. The ASR module, multi-domain NLU engine, and other components of a spoken language processing system may use the hints to more efficiently process input or more accurately generate output. ## Claims ### Claim 1 A system comprising: a computer-readable memory storing executable instructions; and one or more processors in communication with the computer-readable memory, wherein the one or more processors are programmed by the executable instructions to at least: receive text corresponding to a request of a user; process the text in a first natural language understanding (“NLU”) module to generate a first interpretation of the text, and in a second NLU module to generate a second interpretation of the text, wherein the first NLU module is associated with a first domain, and the second NLU module is associated with a second domain, wherein the first interpretation is associated with a first score indicative of a confidence in the first interpretation, and wherein the second interpretation is associated with a second score indicative of a confidence in the second interpretation; select the first interpretation based at least partly on the first score and the second score; and generate a response based at least partly on the first interpretation. ### Claim 2 (depends on 1) The system of claim 1, wherein the executable instructions that program the one or more processors to process the text in the first NLU module comprise executable instructions to: separate at least a portion of the text into a plurality of tokens, wherein the text comprises a plurality of words, and wherein individual tokens of the plurality of tokens comprise one or more words of the of the plurality of words; identify at least a first token of the plurality of tokens as corresponding to an executable command; and identify at least a second token of the plurality of tokens as a parameter of the executable command. ### Claim 3 (depends on 1) The system of claim 1, wherein the first interpretation comprises a plurality of named entities, wherein individual named entities of the plurality of named entities correspond to one or more tokens of the text, and wherein at least one of the plurality of named entities corresponds to an executable command. ### Claim 4 (depends on 1) The system of claim 1, wherein the first NLU module is configured to generate interpretations of text as comprising requests related to the first domain, and wherein the first domain comprises requests related to phone dialing, shopping, getting directions, playing music, or performing a search. ### Claim 5 (depends on 1) The system of claim 1, wherein the confidence in the first interpretation comprises a likelihood that the first interpretation is correct. ### Claim 6 (depends on 1) The system of claim 1, wherein the confidence in the first interpretation is based on whether an executable command corresponds to the first interpretation. ### Claim 7 (depends on 1) The system of claim 1, wherein the executable instructions further program the one or more processors to rank the first interpretation and the second interpretation using the first score and the second score, wherein the first interpretation is ranked higher than the second interpretation, and wherein selecting the first interpretation comprises selecting a highest-ranked interpretation of the text. ### Claim 8 (depends on 1) The system of claim 1, wherein the executable instructions further program the one or more processors to at least: store hint information regarding the response, wherein the response comprises a prompt for additional information regarding the request, and wherein the hint information indicates that a subsequent text will likely relate to the additional information; receive the subsequent text in response to the prompt; process the subsequent text in the first NLU module to generate a third interpretation of the subsequent text, and in the second NLU module generate a fourth interpretation of the subsequent text, wherein the third interpretation is associated with a third score indicative of a confidence in the third interpretation, and wherein the fourth interpretation is associated with a fourth score indicative of a confidence in the fourth interpretation; determine that the third interpretation corresponds to the additional information indicated by the hint information; and increase the third score. ### Claim 9 A computer-implemented method comprising: under control of one or more computing devices configured with specific computer-executable instructions, receiving text corresponding to a request of a user; processing the text in a first natural language understanding (“NLU”) module to generate a first interpretation of the text, and in a second NLU module to generate a second interpretation of the text, wherein the first NLU module is associated with a first domain, and the second NLU module is associated with a second domain, wherein the first interpretation is associated with a first score indicative of a confidence in the first interpretation, and wherein the second interpretation is associated with a second score indicative of a confidence in the second interpretation; selecting the first interpretation based at least partly on the first score and the second score; and generating a response based at least partly on the first interpretation. ### Claim 10 (depends on 9) The computer-implemented method of claim 9, wherein processing the text in the first NLU module to generate the first interpretation comprises: separating at least a portion of the text into a plurality of tokens, wherein the text comprises a plurality of words, and wherein individual tokens of the plurality of tokens comprise one or more words of the of the plurality of words; identifying at least a first token of the plurality of tokens as corresponding to an executable command; and identifying at least a second token of the plurality of tokens as a parameter of the executable command. ### Claim 11 (depends on 9) The computer-implemented method of claim 9, wherein the first interpretation comprises a plurality of named entities, wherein individual named entities of the plurality of named entities correspond to one or more tokens of the text, and wherein at least one of the plurality of named entities corresponds to an executable command. ### Claim 12 (depends on 9) The computer-implemented method of claim 9, wherein the first NLU module is configured to generate interpretations of text as comprising requests related to the first domain, and wherein the first domain comprises requests related to phone dialing, shopping, getting directions, playing music, or performing a search. ### Claim 13 (depends on 9) The computer-implemented method of claim 9, wherein the confidence in the first interpretation comprises a likelihood that the first interpretation is correct. ### Claim 14 (depends on 9) The computer-implemented method of claim 9, wherein the confidence in the first interpretation is based on whether an executable command corresponds to the first interpretation. ### Claim 15 (depends on 9) The computer-implemented method of claim 9, further comprising ranking the first interpretation and the second interpretation based at least partly the first score and the second score, wherein the first interpretation is ranked higher than the second interpretation, and wherein selecting the first interpretation comprises selecting a highest-ranked interpretation of the text. ### Claim 16 (depends on 9) The computer-implemented method of claim 9, further comprising: storing hint information regarding the response, wherein the response comprises a prompt for additional information regarding the request, and wherein the hint information indicates that a subsequent text will likely relate to the additional information; receiving the subsequent text in response to the prompt; processing the subsequent text in the first NLU module to generate a third interpretation of the subsequent text, and in the second NLU module generate a fourth interpretation of the subsequent text, wherein the third interpretation is associated with a third score indicative of a confidence in the third interpretation, and wherein the fourth interpretation is associated with a fourth score indicative of a confidence in the fourth interpretation; determining that the third interpretation corresponds to the additional information indicated by the hint information; and increasing the third score. ### Claim 17 Non-transitory computer readable storage storing executable code that, when executed, causes one or more computing devices to perform a process comprising: receiving text corresponding to a request of a user; processing the text in a first natural language understanding (“NLU”) module to generate a first interpretation of the text, and in a second NLU module to generate a second interpretation of the text, wherein the first NLU module is associated with a first domain, and the second NLU module is associated with a second domain, wherein the first interpretation is associated with a first score indicative of a confidence in the first interpretation, and wherein the second interpretation is associated with a second score indicative of a confidence in the second interpretation; selecting the first interpretation based at least partly on the first score and the second score; and generating a response based at least partly on the first interpretation. ### Claim 18 (depends on 17) The non-transitory computer readable storage of claim 17, wherein the first interpretation comprises a plurality of named entities, wherein individual named entities of the plurality of named entities correspond to one or more tokens of the text, and wherein at least one of the plurality of named entities corresponds to an executable command. ### Claim 19 (depends on 17) The non-transitory computer readable storage of claim 17, wherein the confidence in the first interpretation comprises a likelihood that the first interpretation is correct. ### Claim 20 (depends on 17) The non-transitory computer readable storage of claim 17, wherein the confidence in the first interpretation is based on whether an executable command corresponds to the first interpretation.