# US 10,770,094 B2 · Routing audio streams based on semantically generated result sets > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,770,094 B2 - Document ID: US10770094B2 - Status: Granted - Kind: B2 - Application number: 15/865,428 - Filed: 2018-01-09 - Published: 2019-02-07 - Granted: 2020-09-08 - Earliest claimed: 2018-01-09 - Assignees: Intel IP Corporation - Applicants: Intel IP Corporation - Inventors: Munir Nikolai Alexander Georges; Jakub Nowicki - Web page: https://patentgrep.com/patent/US10770094B2 ## CPC - G10L 25/72: for transmitting results of analysis - G06F 40/30: Semantic analysis - G10L 15/005: {Language recognition} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 25/30: using neural networks - G10L 15/16: using artificial neural networks - G10L 15/28: Constructional details of speech recognition systems - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G10L 2015/225: {Feedback of the input speech} - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) ## Abstract An example apparatus for routing audio streams includes an audio receiver to receive audio from a microphone. The apparatus also includes a classifier to semantically generate a result set based on the audio. The apparatus further includes a scheduler to select a spoken language understanding (SLU) engine based on the result set. The apparatus includes a router to route the audio to the selected SLU engine. ## Claims ### Claim 1 An apparatus for routing audio streams, comprising: an audio receiver to receive audio from a microphone; a classifier to semantically generate a result set comprising a confidence score generated for each of a plurality of different spoken language understanding (SLU) engines based on the audio, wherein the classifier is trained directly on domain data associated with the plurality of different SLU engines associated with different devices; a scheduler to select an SLU engine of the plurality of different SLU engines based on a highest confidence score in the result set; and a router to stream the audio to the selected SLU engine. ### Claim 2 (depends on 1) The apparatus of claim 1, wherein the SLU engine is one of a plurality of SLU engines, each SLU engine associated with a different smart appliance. ### Claim 3 (depends on 1) The apparatus of claim 1, wherein the selected SLU engine is to receive the streamed audio, generate an action based on the streamed audio, and send the action to a smart appliance associated with the selected SLU engine. ### Claim 4 (depends on 1) The apparatus of claim 1, wherein the result set comprises a set of confidence scores calculated for a list of a plurality of SLU engines based on a chunk of the audio. ### Claim 5 (depends on 1) The apparatus of claim 1, wherein the scheduler is to further receive feedback from an application associated with the selected SLU engine and route the audio to a second SLU engine in response to receiving the feedback from the application. ### Claim 6 (depends on 1) The apparatus of claim 1, comprising a chunk generator to generate a plurality of chunks based on the received audio, wherein the classifier is to generate a result set for each of the plurality of chunks, the scheduler is to select an SLU engine for each of the plurality of chunks based on the result set associated with each of the chunks, and the router is to stream each of the chunks to the selected SLU engine for each of the chunks. ### Claim 7 (depends on 1) The apparatus of claim 1, wherein the audio receiver is to receive audio from the microphone in response to detection of a key phrase at a wake-on-voice engine. ### Claim 8 (depends on 1) The apparatus of claim 1, wherein the classifier comprises a neural network that is trained directly on audio using received training data comprising commands for different speech-controlled devices. ### Claim 9 (depends on 1) The apparatus of claim 1, wherein the audio receiver is to cause a display of a prompt comprising privacy options and receive user feedback comprising a consent to enable use of a cloud SLU engine, wherein the selected SLU engine comprises the cloud SLU engine. ### Claim 10 (depends on 9) The apparatus of claim 9, wherein the audio receiver is to continuously receive audio from the microphone in response to receiving the user feedback comprising the consent. ### Claim 11 A method for routing audio streams, comprising: receiving, via a processor, audio from a microphone; semantically generating, via the processor, a result set comprising a confidence score generated for each of a plurality of different spoken language understanding (SLU) engines based on the audio via a classifier trained directly on domain data associated with the plurality of different SLU engines associated with different devices; selecting, via the processor, an SLU engine of the plurality of different SLU engines based on a highest confidence score in the result set; and streaming, via the processor, the audio to the selected SLU engine. ### Claim 12 (depends on 11) The method of claim 11, wherein semantically generating the result set comprises processing the audio using a neural network that is trained directly on audio using received training data comprising commands for different speech-controlled devices. ### Claim 13 (depends on 11) The method of claim 11, comprising generating, at the selected SLU engine, an action based on the streamed audio, and sending the action to a smart appliance associated with the selected SLU engine. ### Claim 14 (depends on 11) The method of claim 11, comprising modifying, via the processor, the classifier at runtime based on a user input. ### Claim 15 (depends on 11) The method of claim 11, comprising receiving, via the processor, feedback from an application of a device associated with the selected SLU engine, and routing the audio to a second SLU engine in response to receiving the feedback from the application. ### Claim 16 (depends on 11) The method of claim 11, comprising generating, via the processor, a plurality of chunks based on the received audio, wherein generating the result set comprises generating a result set for each of the plurality of chunks, wherein selecting the SLU engine comprises selecting the SLU engine for a chunk of the plurality of chunks based on the result set associated with the chunk, and wherein routing the audio comprises routing the chunk to the selected SLU engine. ### Claim 17 (depends on 11) The method of claim 11, comprising receiving, via the processor, audio from the microphone in response to detection of a key phrase at a wake-on-voice engine. ### Claim 18 (depends on 11) The method of claim 11, comprising training, via the processor, the classifier based on user responses to prompts to detect a preference for sending the audio to a local SLU engine or a cloud-based SLU engine. ### Claim 19 (depends on 11) The method of claim 11, comprising displaying a prompt comprising privacy options, and receiving user feedback comprising a consent to send the audio to a cloud SLU engine. ### Claim 20 (depends on 19) The method of claim 19, comprising continuously receiving, via the processor, audio from the microphone in response to receiving the user feedback comprising the consent. ### Claim 21 At least one non-transitory computer readable medium for routing audio streams having instructions stored therein that, in response to being executed on a computing device, cause the computing device to: receive audio from a microphone; semantically generate, via a classifier trained directly on domain data associated with a plurality of different spoken language understanding (SLU) engines associated with different devices, a result set comprising a confidence score generated for each of the plurality of different SLU engines based on the audio; select an SLU engine of the plurality of different SLU engines based on a highest confidence score in the result set; and stream the audio to the selected SLU engine. ### Claim 22 (depends on 21) The at least one non-transitory computer readable medium of claim 21, comprising instructions to process the audio using a neural network that is trained directly on audio using received training data comprising commands for different speech-controlled devices. ### Claim 23 (depends on 21) The at least one non-transitory computer readable medium of claim 21, comprising instructions to generate an action based on the streamed audio and send the action to a smart appliance associated with the selected SLU engine. ### Claim 24 (depends on 21) The at least one non-transitory computer readable medium of claim 21, comprising instructions to modify the classifier at runtime based on a user input. ### Claim 25 (depends on 21) The at least one non-transitory computer readable medium of claim 21, comprising instructions to receive feedback from an application of a device associated with the selected SLU engine and route the audio to a second SLU engine in response to receiving the feedback from application.