# US 10,699,717 B2 · Intelligent assistant for home automation > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 10,699,717 B2 - Document ID: US10699717B2 - Status: Granted - Kind: B2 - Application number: 16/175,208 - Filed: 2018-10-30 - Published: 2019-03-07 - Granted: 2020-06-30 - Earliest claimed: 2014-05-30 - Assignees: Apple Inc. - Applicants: Apple Inc. - Inventors: Ryan M. Orr; Garett R. Nell; Benjamin L. Brumbaugh - Web page: https://patentgrep.com/patent/US10699717B2 ## CPC - G10L 17/22: Interactive procedures; Man-machine interfaces - G05B 15/02: electric - G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback} - H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)} - H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)} - G05B 2219/23373: Interactive guidance by voice message - G05B 2219/23386: Voice, vocal command or message - G05B 2219/2642: Domotique, domestic, home control, automation, smart house - G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING - G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G) - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device. ## Claims ### Claim 1 A user device comprising: one or more processors; memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for: receiving an audio input representing a user request for an electronic device to perform an action, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the electronic device for performing the action; determining one or more possible electronic device characteristics corresponding to the audio input; retrieving a data structure representing a set of electronic devices of a predetermined location; determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input; determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent. ### Claim 2 (depends on 1) The user device of claim 1, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action. ### Claim 3 (depends on 1) The user device of claim 1, wherein the one or more possible electronic device characteristics comprise one or more words associated with electronic devices that are controllable by the user device, or associated with one or more states corresponding to the electronic devices that are controllable by the user device. ### Claim 4 (depends on 1) The user device of claim 1, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices. ### Claim 5 (depends on 4) The user device of claim 4, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices. ### Claim 6 (depends on 1) The user device of claim 1, wherein the data structure defines a current state of each electronic device in the set of electronic devices. ### Claim 7 (depends on 6) The user device of claim 6, wherein the one or more programs comprise further instructions for: determining the user intent further based on the current state of each of the one or more candidate electronic devices. ### Claim 8 (depends on 6) The user device of claim 6 wherein the one or more programs comprise further instructions for: after providing the command, receiving an updated state from the electronic device; and updating the current state of the electronic device based on the updated state received from the electronic device. ### Claim 9 (depends on 8) The user device of claim 8, wherein the one or more programs comprise further instructions for: outputting an indication of a result of the command based on the updated state of the electronic device received from the electronic device. ### Claim 10 (depends on 9) The user device of claim 9, wherein the indication of the result of the command is an audio indication, a visual indication, or both an audio and a visual indication. ### Claim 11 (depends on 1) The user device of claim 1, wherein the one or more programs comprise further instructions for: determining a location corresponding to the audio input; and determining, based on the determined location, the predetermined location from a plurality of predetermined locations. ### Claim 12 (depends on 1) The user device of claim 1, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises: determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms. ### Claim 13 (depends on 1) The user device of claim 1, wherein determining the user intent further comprises: determining the electronic device from the one or more candidate electronic devices based on the one or more possible electronic device characteristics and the one or more actual electronic device characteristics of the one or more candidate electronic devices. ### Claim 14 (depends on 1) The user device of claim 1, wherein the one or more programs comprise further instructions for: determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location. ### Claim 15 A method for controlling an electronic device using a virtual assistant on a user device having one or more processors and a memory, the method comprising: at the user device: receiving an audio input representing a user request for an electronic device to perform an action, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the electronic device for performing the action; determining one or more possible electronic device characteristics corresponding to the audio input; retrieving a data structure representing a set of electronic devices of a predetermined location; determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input; determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent. ### Claim 16 (depends on 15) The method of claim 15, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action. ### Claim 17 (depends on 15) The method of claim 15, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices. ### Claim 18 (depends on 17) The method of claim 17, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices. ### Claim 19 (depends on 15) The method of claim 15, further comprising: determining a location corresponding to the audio input; and determining, based on the determined location, the predetermined location from a plurality of predetermined locations. ### Claim 20 (depends on 15) The method of claim 15, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises: determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms. ### Claim 21 (depends on 15) The method of claim 15, further comprising: determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location. ### Claim 22 A non-transitory computer-readable storage medium comprising instructions for one or more programs for execution by one or more processors of a user device, the one or more programs including instructions which, when executed by the one or more processors, cause the user device to perform the method of: receiving an audio input representing a user request for an electronic device to perform an action, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the electronic device for performing the action; determining one or more possible electronic device characteristics corresponding to the audio input; retrieving a data structure representing a set of electronic devices of a predetermined location; determining, based on the data structure, one or more candidate electronic devices from the set of electronic devices, the one or more candidate electronic devices corresponding to the audio input; determining, based on the one or more possible electronic device characteristics and one or more actual electronic device characteristics of the one or more candidate electronic devices, a user intent corresponding to the audio input; and providing a command that causes an electronic device of the one or more candidate electronic devices to perform the action corresponding to the user intent. ### Claim 23 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein the audio input is in natural language form, and wherein the audio input is ambiguous with respect to defining the action. ### Claim 24 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein the data structure defines one or more actual device characteristics associated with each electronic device in the set of electronic devices. ### Claim 25 (depends on 24) The non-transitory computer-readable storage medium of claim 24, wherein the one or more actual electronic device characteristics comprise one or more words associated with each electronic device in the set of electronic devices, or associated with one or more states corresponding to each electronic device in the set of electronic devices. ### Claim 26 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs comprise further instructions for: determining a location corresponding to the audio input; and determining, based on the determined location, the predetermined location from a plurality of predetermined locations. ### Claim 27 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein determining the one or more candidate electronic devices from the set of electronic devices comprises: determining one or more alternative terms associated with a term in the audio input, wherein the one or more candidate electronic devices is determined based on the one or more alternative terms. ### Claim 28 (depends on 22) The non-transitory computer-readable storage medium of claim 22, wherein the one or more programs comprise further instructions for: determining a relative location of the user device with respect to an area within the predetermined location, wherein the user intent is determined based on the relative location.