# US 2020/0286493 A1 · Intelligent assistant for home automation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286493 A1
- Document ID: US20200286493A1
- Status: Published application
- Kind: A1
- Application number: 16/881,625
- Filed: 2020-05-22
- Published: 2020-09-10
- Earliest claimed: 2014-05-30
- Assignees: Apple Inc.
- Applicants: Apple Inc.
- Inventors: Ryan M. ORR; Garett R. NELL; Benjamin L. BRUMBAUGH
- Web page: https://patentgrep.com/patent/US20200286493A1

## CPC

- G10L 17/22: Interactive procedures; Man-machine interfaces
- G05B 15/02: electric
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)}
- H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)}
- G05B 2219/23373: Interactive guidance by voice message
- G05B 2219/23386: Voice, vocal command or message
- G05B 2219/2642: Domotique, domestic, home control, automation, smart house
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G05B: CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS (systems for controlling or regulating non-electric variables G05D; systems for regulating electric or magnetic variables G05F; control devices or systems insofar as characterised by mechanical features only G05G)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)

## Abstract

This relates to systems and processes for using a virtual assistant to control electronic devices. In one example process, a user can speak an input in natural language form to a user device to control one or more electronic devices. The user device can transmit the user speech to a server to be converted into a textual representation. The server can identify the one or more electronic devices and appropriate commands to be performed by the one or more electronic devices based on the textual representation. The identified one or more devices and commands to be performed can be transmitted back to the user device, which can forward the commands to the appropriate one or more electronic devices for execution. In response to receiving the commands, the one or more electronic devices can perform the commands and transmit their current states to the user device.

## Claims

### Claim 1

A non-transitory computer-readable storage medium storing one or more programs, the one or more programs comprising instructions, which when executed by one or more processors of one or more servers, cause the one or more servers to:
receive, from a user device, data corresponding to an audio input comprising a user speech;
perform speech to text conversion on the data corresponding to the audio input to generate a textual representation of the user speech;
determine that the textual representation of the user speech represents a user intent to change the state of each of a plurality of electronic devices based on a stored configuration, wherein the stored configuration defines a state of each of the plurality of electronic devices to use in response to a command that references the configuration; and
transmit to the user device:
a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration; and
identifications associated with each of the plurality of commands, wherein the identifications identify each of the plurality of electronic devices for performing each of the plurality of commands.

### Claim 2 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the one or more servers to:
after transmitting the plurality of commands and the identifications, receive an updated state of each of the plurality of electronic devices from the user device; and
update a state of one or more of the plurality of electronic devices in a database based on the received updated state of each of the plurality of electronic devices.

### Claim 3 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the one or more servers to:
after receiving the updated states of each of the plurality of electronic devices, generate an audio or visual indication of a result of the transmitted plurality of commands, wherein the result is based on the transmitted plurality of commands and the received updated state of each of the plurality of electronic devices; and
transmit, to the user device, data corresponding to the audio or visual indication of the result.

### Claim 4 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein at least one of the updated states of each of the plurality of electronic devices is an error state, and wherein the error state represents an unavailable or undetermined state of an electronic device.

### Claim 5 (depends on 2)

The non-transitory computer-readable storage medium of claim 2, wherein the updated state of each of the plurality of electronic devices comprises at least one or an ON/OFF state, a dimmable state, a color state, an ACTIVE/INACTIVE state, a LOCKED/UNLOCKED state, an OPEN/CLOSED state, or a temperature state.

### Claim 6 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the user device is a mobile phone, a desktop computer, a laptop computer, a tablet computer, a portable media player, a television, a television set-top box, or a wearable electronic device.

### Claim 7 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the plurality of electronic devices includes at least one of a light bulb, an electrical outlet, a switch, a door lock, a garage door, and a thermostat.

### Claim 8 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the one or more servers to:
receive, from the user device, data corresponding to a location of the user device.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the one or more servers determine that the textual representation of the user speech represents a user intent to change the state of each of a plurality of electronic devices based on a stored configuration further based on the data corresponding to the location of the user device.

### Claim 10 (depends on 1)

The non-transitory computer-readable storage medium of claim 1, wherein the one or more programs include further instructions, which when executed by the one or more processors, cause the one or more servers to:
receive, from the user device, data corresponding to a current date and time.

### Claim 11 (depends on 10)

The non-transitory computer-readable storage medium of claim 10, wherein the one or more servers determine that the textual representation of the user speech represents a user intent to change the state of each of a plurality of electronic devices based on a stored configuration further based on the data corresponding to the current date and time.

### Claim 12

A method for controlling electronic devices using a virtual assistant on a user device, the method comprising:
receiving, by one or more servers, data corresponding to an audio input comprising a user speech;
performing speech to text conversion on the data corresponding to the audio input to generate a textual representation of the user speech;
determining that the textual representation of the user speech represents a user intent to change the state of each of a plurality of electronic devices based on a stored configuration, wherein the stored configuration defines a state of each of the plurality of electronic devices to use in response to a command that references the configuration; and
transmitting to the user device:
a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration; and
identifications associated with each of the plurality of commands, wherein the identifications identify each of the plurality of electronic devices for performing each of the plurality of commands.

### Claim 13

A system, comprising:
one or more processors;
a memory; and
one or more programs, wherein the one or more programs are stored on the memory, and wherein the one or more programs include instructions for:
receiving, from a user device, data corresponding to an audio input comprising a user speech;
performing speech to text conversion on the data corresponding to the audio input to generate a textual representation of the user speech;
determining that the textual representation of the user speech represents a user intent to change the state of each of a plurality of electronic devices based on a stored configuration, wherein the stored configuration defines a state of each of the plurality of electronic devices to use in response to a command that references the configuration; and
transmitting to the user device:
a plurality of commands to set the state of each of the plurality of electronic devices based on the configuration; and
identifications associated with each of the plurality of commands, wherein the identifications identify each of the plurality of electronic devices for performing each of the plurality of commands.
