# US 12,118,273 B2 · Local voice data processing > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,118,273 B2 - Document ID: US12118273B2 - Status: Granted - Kind: B2 - Application number: 18/154,228 - Filed: 2023-01-13 - Published: 2023-09-14 - Granted: 2024-10-15 - Earliest claimed: 2020-01-31 - Assignees: Sonos, Inc. - Applicants: Sonos, Inc. - Inventors: Sebastien Maury; Joseph Dureau; Thibaut Lorrain; Do Kyun Kim - Web page: https://patentgrep.com/patent/US12118273B2 ## CPC - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - G06F 3/165: {Management of the audio stream, e.g. setting of volume, audio stream path} - G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue - G10L 17/02: Preprocessing operations, e.g. segment selection; Pattern representation or modelling, e.g. based on linear discriminant analysis [LDA] or principal components; Feature selection or extraction - G10L 17/06: Decision making techniques; Pattern matching strategies - G10L 2015/088: {Word spotting} - G10L 2015/223: {Execution procedure of a spoken command} - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING ## Abstract Example techniques relate to local voice control in a media playback system. A satellite device (e.g., a playback device or microcontroller unit) may be configured to recognize a local set of keywords in voice inputs including context specific keywords (e.g., for controlling an associated smart device) as well as keywords corresponding to a subset of media playback commands for controlling playback devices in the media playback system. The satellite device may fall back to a hub device (e.g., a playback device) configured to recognize a more extensive set of keywords. In some examples, either device may fall back to the cloud for processing of other voice inputs. ## Claims ### Claim 1 A system comprising: a microcontroller unit (MCU); a network interface; at least one processor; and at least one non-transitory computer-readable medium comprising program instructions that are executable by the at least one processor such that the system is configured to: store, in data storage of the MCU, data representing local keywords corresponding to smart Internet-of-Things (IoT) commands; capture an input sound data stream from one or more microphones; transmit, via the network interface over a local area network to a hub device, data representing the input sound data stream for voice processing of one or more first voice inputs in the input sound data stream into respective commands by a voice assistant of the hub device, wherein the MCU and the hub device are connected to the local area network; monitor the captured input sound data stream for the local keywords corresponding the smart IoT commands; detect at least one keyword of the local keywords in a portion of the captured input sound data stream comprising a second voice input; determine at least one particular smart IoT command corresponding to the at least one keyword; send, via the network interface to the hub device, an indication that the MCU is processing the portion of the captured input sound data stream comprising the second voice input, wherein the hub device forgoes processing of the portion of the captured input sound data stream comprising the second voice input when the indication is received; and cause a smart IoT device to carry out the at least one particular smart IoT command. ### Claim 2 (depends on 1) The system of claim 1, wherein the smart IoT device comprises the MCU. ### Claim 3 (depends on 2) The system of claim 2, wherein the smart IoT device comprises a smart appliance. ### Claim 4 (depends on 3) The system of claim 3, wherein the program instructions that are executable by the at least one processor such that the system is configured to cause the smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the system is configured to: send, via the network interface, instructions to cause the smart appliance to turn on. ### Claim 5 (depends on 1) The system of claim 1, wherein the program instructions that are executable by the at least one processor such that the system is configured to send the indication that the MCU is processing the portion of the captured input sound data stream comprising the second voice input comprise program instructions that are executable by the at least one processor such that the system is configured to: send, via the network interface, instructions to forgo voice processing of the portion of the captured input sound data stream comprising the second voice input. ### Claim 6 (depends on 1) The system of claim 1, wherein the smart IoT device is grouped with at least one additional smart IoT device in a group, and wherein the program instructions that are executable by the at least one processor such that the system is configured to cause the smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the system is configured to: send, via the network interface, instructions to cause the smart IoT device and the at least one additional smart IoT device to carry out the at least one particular smart IoT command. ### Claim 7 (depends on 6) The system of claim 6, wherein the smart IoT device comprises a first playback device, wherein the at least one additional smart IoT device comprises at least one second playback device, and wherein the program instructions that are executable by the at least one processor such that the system is configured to send the instructions to cause the smart IoT device and the at least one additional smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the system is configured to: send, via the network interface, instructions to cause the first playback device and the at least one second playback device to play back audio content in synchrony. ### Claim 8 (depends on 6) The system of claim 6, wherein the smart IoT device comprises a first illumination device, wherein the at least one additional smart IoT device comprises at least one second illumination device, and wherein the program instructions that are executable by the at least one processor such that the system is configured to send the instructions to cause the smart IoT device and the at least one additional smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the system is configured to: send, via the network interface, instructions to cause the first illumination device and the at least one second illumination device to toggle an illumination state. ### Claim 9 (depends on 1) The system of claim 1, wherein the system further comprises the hub device, and wherein the at least one non-transitory computer-readable medium further comprises program instructions that are executable by the at least one processor such that the system is configured to: determine, via the voice assistant of the hub device, respective intents of the one or more first voice inputs. ### Claim 10 (depends on 1) The system of claim 1, wherein the one or more first voice inputs exclude the local keywords, and wherein the voice assistant of the hub device is configured to process voice inputs comprising additional keywords relative to the local keywords stored in the data representing the local keywords. ### Claim 11 A microcontroller unit (MCU) comprising: a network interface; at least one processor; and at least one non-transitory computer-readable medium comprising program instructions that are executable by the at least one processor such that the MCU is configured to: store, in data storage of the MCU, data representing local keywords corresponding to smart Internet-of-Things (IoT) commands; capture an input sound data stream from one or more microphones; transmit, via the network interface over a local area network to a hub device, data representing the input sound data stream for voice processing of one or more first voice inputs in the input sound data stream into respective commands by a voice assistant of the hub device, wherein the MCU and the hub device are connected to the local area network; monitor the captured input sound data stream for the local keywords corresponding the smart IoT commands; detect at least one keyword of the local keywords in a portion of the captured input sound data stream comprising a second voice input; determine at least one particular smart IoT command corresponding to the at least one keyword; send, via the network interface to the hub device, an indication that the MCU is processing the portion of the captured input sound data stream comprising the second voice input, wherein the hub device forgoes processing of the portion of the captured input sound data stream comprising the second voice input when the indication is received; and cause a smart IoT device to carry out the at least one particular smart IoT command. ### Claim 12 (depends on 11) The MCU of claim 11, wherein the MCU is configured to be carried in a housing of the smart IoT device. ### Claim 13 (depends on 12) The MCU of claim 12, wherein the smart IoT device comprises a smart appliance. ### Claim 14 (depends on 13) The MCU of claim 13, wherein the program instructions that are executable by the at least one processor such that the MCU is configured to cause the smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the MCU is configured to: send, via the network interface, instructions to cause the smart appliance to turn on. ### Claim 15 (depends on 11) The MCU of claim 11, wherein the program instructions that are executable by the at least one processor such that the MCU is configured to send the indication that the MCU is processing the portion of the captured input sound data stream comprising the second voice input comprise program instructions that are executable by the at least one processor such that the MCU is configured to: send, via the network interface, instructions to forgo voice processing of the portion of the captured input sound data stream comprising the second voice input. ### Claim 16 (depends on 11) The MCU of claim 11, wherein the smart IoT device is grouped with at least one additional smart IoT device in a group, and wherein the program instructions that are executable by the at least one processor such that the MCU is configured to cause the smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the MCU is configured to: send, via the network interface, instructions to cause the smart IoT device and the at least one additional smart IoT device to carry out the at least one particular smart IoT command. ### Claim 17 (depends on 16) The MCU of claim 16, wherein the smart IOT device comprises a first playback device, wherein the at least one additional smart IoT device comprises at least one second playback device, and wherein the program instructions that are executable by the at least one processor such that the MCU is configured to send the instructions to cause the smart IoT device and the at least one additional smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the MCU is configured to: send, via the network interface, instructions to cause the first playback device and the at least one second playback device to play back audio content in synchrony. ### Claim 18 (depends on 16) The MCU of claim 16, wherein the smart IOT device comprises a first illumination device, wherein the at least one additional smart IoT device comprises at least one second illumination device, and wherein the program instructions that are executable by the at least one processor such that the MCU is configured to send the instructions to cause the smart IOT device and the at least one additional smart IoT device to carry out the at least one particular smart IoT command comprise program instructions that are executable by the at least one processor such that the MCU is configured to: send, via the network interface, instructions to cause the first illumination device and the at least one second illumination device to toggle an illumination state. ### Claim 19 (depends on 11) The MCU of claim 11, wherein the one or more first voice inputs exclude the local keywords, and wherein the voice assistant of the hub device is configured to process voice inputs comprising additional keywords relative to the local keywords stored in the data representing the local keywords. ### Claim 20 At least one non-transitory computer-readable medium comprising program instructions that are executable by at least one processor such that a microcontroller unit (MCU) is configured to: store, in data storage of the MCU, data representing local keywords corresponding to smart Internet-of-Things (IoT) commands; capture an input sound data stream from one or more microphones; transmit, via a network interface over a local area network to a hub device, data representing the input sound data stream for voice processing of one or more first voice inputs in the input sound data stream into respective commands by a voice assistant of the hub device; monitor the captured input sound data stream for the local keywords corresponding the smart IoT commands; detect at least one keyword of the local keywords in a portion of the captured input sound data stream comprising a second voice input; determine at least one particular smart IoT command corresponding to the at least one keyword; send, via the network interface to the hub device, an indication that the MCU is processing the portion of the captured input sound data stream comprising the second voice input, wherein the hub device forgoes processing of the portion of the captured input sound data stream comprising the second voice input when the indication is received; and cause a smart IoT device to carry out the at least one particular smart IoT command.