# US 10,170,123 B2 · Intelligent assistant for home automation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,170,123 B2 - Document ID: US10170123B2 - Status: Granted - Kind: B2 - Application number: 14/503,105 - Filed: 2014-09-30 - Published: 2015-12-03 - Granted: 2019-01-01 - Earliest claimed: 2014-05-30 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Ryan M. Orr; Garett R. Nell; Benjamin L. Brumbaugh - Web page: https://patentgrep.com/patent/US10170123B2 ## CPC - G10L 17/22: Interactive procedures; Man-machine interfaces - G05B 15/02: electric - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)} - H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)} - G05B 2219/23373: Interactive guidance by voice message - G05B 2219/23386: Voice, vocal command or message - G05B 2219/2642: Domotique, domestic, home control, automation, smart house - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device. ## Claims ### Claim 1 A method for controlling electronic devices using a virtual assistant on a user device having a memory and at least one processor, the method comprising: at the user device: receiving, an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving, from the one or more servers: an identification of a first electronic device determined by the one or more servers based on the data corresponding to the audio input; and a first command to be performed by the first electronic device determined by the one or more servers based on the data corresponding to the audio input; transmitting the first command to the first electronic device; receiving, after transmitting the first command to the first electronic device, a second audio input; transmitting a state query to the first electronic device in accordance with the second audio input; receiving, from the first electronic device, a current state of the first electronic device; and transmitting the current state of the first electronic device to the one or more servers to be stored as at least a portion of a configuration, wherein the configuration defines a plurality of states of a plurality of corresponding electronic devices to use in response to a command that references the configuration. ### Claim 2 (depends on 1) The method of claim 1, further comprising: receiving, after transmitting the first command to the first electronic device and prior to receiving the second audio input, a current state of the first electronic device from the first electronic device; and transmitting the current state of the first electronic device to the one or more servers. ### Claim 3 (depends on 2) The method of claim 2, wherein the first electronic device comprises a light bulb. ### Claim 4 (depends on 3) The method of claim 3, wherein the first command comprises a command to set an ON/OFF state, dimmable state, or color state of the light bulb. ### Claim 5 (depends on 4) The method of claim 4, wherein the current state of the first electronic device comprises the ON/OFF state, dimmable state, or color state of the light bulb after transmitting the first command to set the ON/OFF state, dimmable state, or color state of the light bulb. ### Claim 6 (depends on 2) The method of claim 2, wherein the first electronic device comprises an electrical outlet. ### Claim 7 (depends on 6) The method of claim 6, wherein the first command comprises a command to set an ACTIVE/INACTIVE state of the electrical outlet. ### Claim 8 (depends on 7) The method of claim 7, wherein the current state of the first electronic device comprises the ACTIVE/INACTIVE state of the electrical outlet after transmitting the command to set the ACTIVE/INACTIVE state of the electrical outlet. ### Claim 9 (depends on 2) The method of claim 2, wherein the first electronic device comprises a switch. ### Claim 10 (depends on 9) The method of claim 9, wherein the first command comprises a command to set an ON/OFF state of the switch. ### Claim 11 (depends on 10) The method of claim 10, wherein the current state of the first electronic device comprises the ON/OFF state of the electrical outlet after transmitting the command to set the ON/OFF state of the switch. ### Claim 12 (depends on 2) The method of claim 2, wherein the first electronic device comprises a door lock. ### Claim 13 (depends on 12) The method of claim 12, wherein the first command comprises a command to set a LOCKED/UNLOCKED state of the door lock. ### Claim 14 (depends on 13) The method of claim 13, wherein the current state of the first electronic device comprises the LOCKED/UNLOCKED state of the door lock after transmitting the command to set the LOCKED/UNLOCKED state of the door lock. ### Claim 15 (depends on 2) The method of claim 2, wherein the first electronic device comprises a garage door. ### Claim 16 (depends on 15) The method of claim 15, wherein the first command comprises a command to set an OPEN/CLOSED state of the garage door. ### Claim 17 (depends on 16) The method of claim 16, wherein the current state of the first electronic device comprises the OPEN/CLOSED state of the garage door after transmitting the command to set the OPEN/CLOSED state of the garage door. ### Claim 18 (depends on 2) The method of claim 2, wherein the first electronic device comprises a thermostat. ### Claim 19 (depends on 18) The method of claim 18, wherein the first command comprises a command to set a numerical value of a temperature setting of the thermostat. ### Claim 20 (depends on 19) The method of claim 19, wherein the current state of the first electronic device comprises the numerical value of the temperature setting of the thermostat after transmitting the first command to set the numerical value of the temperature setting of the thermostat. ### Claim 21 (depends on 2) The method of claim 2, further comprising outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied. ### Claim 22 (depends on 2) The method of claim 2, wherein the user device comprises a database for storing a state of each of a plurality of electronic devices, the plurality of electronic devices comprising the first electronic device, and wherein the method further comprises updating a state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device. ### Claim 23 (depends on 2) The method of claim 2, wherein the method further comprises outputting an indication of a result of the first command based on the first command and the current state of the first electronic device received from the first electronic device, wherein the indication of the result is an audio, visual, or both an audio and visual indication. ### Claim 24 (depends on 1) The method of claim 1, wherein the user device comprises a mobile phone, desktop computer, laptop computer, tablet computer, portable media player, television, television set-top box, or wearable electronic device. ### Claim 25 (depends on 1) The method of claim 1, further comprising: receiving, prior to receiving the second audio input, from the one or more servers: an identification of a second electronic device determined by the one or more servers based on the data corresponding to the audio input; and a second command to be performed by the second electronic device determined by the one or more servers based on the data corresponding to the audio input; transmitting the second command to the second electronic device; receiving, after transmitting the second command to the second electronic device, a current state of the second electronic device from the second electronic device; and transmitting the current state of the second electronic device to the one or more servers. ### Claim 26 (depends on 1) The method of claim 1, wherein the user device comprises a microphone. ### Claim 27 (depends on 1) The method of claim 1, wherein transmitting the state query to the first electronic device in accordance with the second audio input comprises: transmitting data corresponding to the second audio input to the one or more servers; receiving, from the one or more servers, an instruction to query a state of the first electronic device determined by the one or more servers based on the data corresponding to the second audio input; and transmitting the state query to the first electronic device based on the instruction to query the state of the first electronic device. ### Claim 28 A non-transitory computer-readable storage medium comprising one or more programs for execution by one or more processors of a user device, the one or more programs including instructions which, when executed by the one or more processors, cause the user device to perform the method of: receiving an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving, from the one or more servers: an identification of a first electronic device determined by the one or more servers based on the data corresponding to the audio input; and a first command to be performed by the first electronic device determined by the one or more servers based on the data corresponding to the audio input; transmitting the first command to the first electronic device; receiving, after transmitting the first command to the first electronic device, a second audio input; transmitting a state query to the first electronic device in accordance with the second audio input; receiving, from the first electronic device, a current state of the first electronic device; and transmitting the current state of the first electronic device to the one or more servers to be stored as at least a portion of a configuration, wherein the configuration defines a plurality of states of a plurality of corresponding electronic devices to use in response to a command that references the configuration. ### Claim 29 (depends on 28) The non-transitory computer-readable storage medium of claim 28, the one or more programs further including instructions for: receiving, after transmitting the first command to the first electronic device and prior to receiving the second audio input, a current state of the first electronic device from the first electronic device; and transmitting the current state of the first electronic device to the one or more servers. ### Claim 30 (depends on 29) The non-transitory computer-readable storage medium of claim 29, the one or more programs further including instructions for: outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied. ### Claim 31 (depends on 28) The non-transitory computer-readable storage medium of claim 28, wherein the user device comprises a microphone. ### Claim 32 A system comprising: one or more processors; memory; one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving an audio input comprising user speech; transmitting data corresponding to the audio input to one or more servers; receiving, from the one or more servers: an identification of a first electronic device determined by the one or more servers based on the data corresponding to the audio input; and a first command to be performed by the first electronic device determined by the one or more servers based on the data corresponding to the audio input; transmitting the first command to the first electronic device; receiving, after transmitting the first command to the first electronic device, a second audio input; transmitting a state query to the first electronic device in accordance with the second audio input; receiving, from the first electronic device, a current state of the first electronic device; and transmitting the current state of the first electronic device to the one or more servers to be stored as at least a portion of a configuration, wherein the configuration defines a plurality of states of a plurality of corresponding electronic devices to use in response to a command that references the configuration. ### Claim 33 (depends on 32) The system of claim 32, further comprising instructions for: receiving, after transmitting the first command to the first electronic device and prior to receiving the second audio input, a current state of the first electronic device from the first electronic device; and transmitting the current state of the first electronic device to the one or more servers. ### Claim 34 (depends on 33) The system of claim 33, further comprising instructions for: outputting a notification associated with the current state of the first electronic device in response to determining that a predetermined condition has been satisfied. ### Claim 35 (depends on 33) The system of claim 33, further comprising a database for storing a state of each of a plurality of electronic devices, the plurality of electronic devices comprising the first electronic device, and wherein the one or more programs further comprises instructions for updating a state of the first electronic device stored in the database based at least in part on the current state of the first electronic device received from the first electronic device. ### Claim 36 (depends on 32) The system of claim 32, further comprising a microphone.