# US 2019/0295542 A1 · Voice-based user interface with dynamically switchable endpoints

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2019/0295542 A1
- Document ID: US20190295542A1
- Status: Published application
- Kind: A1
- Application number: 15/936,109
- Filed: 2018-03-26
- Published: 2019-09-26
- Earliest claimed: 2018-03-26
- Assignees: Midea Group Co., Ltd.
- Applicants: Midea Group Co., Ltd.
- Inventors: Haibin Huang; Chi Zhang; Xiaofeng Xu; Chen Zhang; Dongyan Wang
- Web page: https://patentgrep.com/patent/US20190295542A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- H04L 12/2816: {Controlling appliance services of a home automation network by calling their functionalities (arrangements in telecontrol or telemetry systems for selectively calling a substation from a main station; in which substation desired apparatus is selected for applying a control signal thereto or for obtaining measured values therefrom H04Q9/00)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 2015/223: {Execution procedure of a spoken command}
- H04L 12/2803: {Home automation networks}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

A method and system of controlling a digital assistant with dynamically switchable endpoint devices, comprising: dynamically selecting a respective input endpoint device and a respective controlled device for each of a plurality of voice-based requests from a user to the computing system, including: at a first point in time, acquiring respective instances of a first voice input from a first set of two or more input endpoint devices; obtaining a representative copy of the first voice input based on the respective instances of the first voice input that have been acquired from the first set of two or more input endpoint devices; determining a first actionable intent based on the representative copy of the first voice input; and dispatching a first encoded instruction to a first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the first actionable intent.

## Claims

### Claim 1

A method of controlling a digital assistant with dynamically switchable endpoint devices, comprising:
at a computing system having one or more processors and memory:
registering a plurality of input endpoint devices and a plurality of controlled endpoint devices, wherein each of the plurality of input endpoint devices has a respective audio input interface and is configured to transmit a captured voice input to the computing system over a network, and wherein each of the plurality of controlled endpoint devices is configured to perform a requested task in accordance with a corresponding instruction received from the computing system over the network; and
dynamically selecting a respective input endpoint device and a respective controlled device for each of a plurality of voice-based requests from a user to the computing system, including:
at a first point in time, acquiring respective instances of a first voice input from a first set of two or more input endpoint devices among the plurality of input endpoint devices;
obtaining a representative copy of the first voice input based on the respective instances of the first voice input that have been acquired from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a first actionable intent based on the representative copy of the first voice input; and
dispatching a first encoded instruction to a first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the first actionable intent.

### Claim 2 (depends on 1)

The method of claim 1, including:
registering a plurality of output endpoint devices, wherein each of the plurality of output endpoint devices has a respective audio output interface that is configured to output audio content received from the computing system over the network; and
dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, including:
dispatching a first audio output corresponding to the first actionable intent to a first output endpoint device selected from the plurality of output endpoint devices over the network, wherein the first output endpoint device outputs the first audio output.

### Claim 3 (depends on 2)

The method of claim 2, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
at a second point in time, acquiring respective instances of a second voice input from a second set of two or more input endpoint devices among the plurality of input endpoint devices, wherein the second set of two or more input endpoint devices are distinct from the first set of two or more input endpoint devices;
obtaining a representative copy of the second voice input based on the respective instances of the second voice input that have been acquired from the second set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a second actionable intent based on the representative copy of the second voice input; and
dispatching a second encoded instruction to the first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the second actionable intent.

### Claim 4 (depends on 3)

The method of claim 3, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
dispatching a second audio output corresponding to the second actionable intent to a second output endpoint device selected from the plurality of output endpoint devices over the network, wherein the second output endpoint device is distinct from the first output endpoint device, and wherein the second output endpoint device outputs the first audio output.

### Claim 5 (depends on 2)

The method of claim 2, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
at a third point in time, acquiring respective instances of a third voice input from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
obtaining a representative copy of the third voice input based on the respective instances of the third voice input that have been acquired from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a third actionable intent based on the representative copy of the third voice input; and
dispatching a third encoded instruction to a second controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the third actionable intent, wherein the second controlled endpoint device is distinct from the first controlled endpoint device.

### Claim 6 (depends on 1)

The method of claim 1, wherein the plurality of input endpoint devices and the plurality of controlled endpoint devices include devices located in a respective smart home environment.

### Claim 7 (depends on 1)

The method of claim 1, wherein the first set of two or more input endpoint device includes the first controlled endpoint device and at least one device that is not among the plurality of controlled endpoint devices.

### Claim 8

A computing system, comprising:
one or more processors; and
memory storing instructions, the instructions, when executed by the one or more processors, cause the processors to perform operations comprising:
registering a plurality of input endpoint devices and a plurality of controlled endpoint devices, wherein each of the plurality of input endpoint devices has a respective audio input interface and is configured to transmit a captured voice input to the computing system over a network, and wherein each of the plurality of controlled endpoint devices is configured to perform a requested task in accordance with a corresponding instruction received from the computing system over the network; and
dynamically selecting a respective input endpoint device and a respective controlled device for each of a plurality of voice-based requests from a user to the computing system, including:
at a first point in time, acquiring respective instances of a first voice input from a first set of two or more input endpoint devices among the plurality of input endpoint devices;
obtaining a representative copy of the first voice input based on the respective instances of the first voice input that have been acquired from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a first actionable intent based on the representative copy of the first voice input; and
dispatching a first encoded instruction to a first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the first actionable intent.

### Claim 9 (depends on 8)

The computing system of claim 8, wherein the operations include:
registering a plurality of output endpoint devices, wherein each of the plurality of output endpoint devices has a respective audio output interface that is configured to output audio content received from the computing system over the network; and
dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, including:
dispatching a first audio output corresponding to the first actionable intent to a first output endpoint device selected from the plurality of output endpoint devices over the network, wherein the first output endpoint device outputs the first audio output.

### Claim 10 (depends on 9)

The computing system of claim 9, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
at a second point in time, acquiring respective instances of a second voice input from a second set of two or more input endpoint devices among the plurality of input endpoint devices, wherein the second set of two or more input endpoint devices are distinct from the first set of two or more input endpoint devices;
obtaining a representative copy of the second voice input based on the respective instances of the second voice input that have been acquired from the second set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a second actionable intent based on the representative copy of the second voice input; and
dispatching a second encoded instruction to the first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the second actionable intent.

### Claim 11 (depends on 10)

The computing system of claim 10, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
dispatching a second audio output corresponding to the second actionable intent to a second output endpoint device selected from the plurality of output endpoint devices over the network, wherein the second output endpoint device is distinct from the first output endpoint device, and wherein the second output endpoint device outputs the first audio output.

### Claim 12 (depends on 9)

The computing system of claim 9, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
at a third point in time, acquiring respective instances of a third voice input from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
obtaining a representative copy of the third voice input based on the respective instances of the third voice input that have been acquired from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a third actionable intent based on the representative copy of the third voice input; and
dispatching a third encoded instruction to a second controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the third actionable intent, wherein the second controlled endpoint device is distinct from the first controlled endpoint device.

### Claim 13 (depends on 8)

The computing system of claim 8, wherein the plurality of input endpoint devices and the plurality of controlled endpoint devices include devices located in a respective smart home environment.

### Claim 14 (depends on 8)

The computing system of claim 8, wherein the first set of two or more input endpoint device includes the first controlled endpoint device and at least one device that is not among the plurality of controlled endpoint devices.

### Claim 15

A non-transitory computer-readable storage medium storing instructions, the instructions, when executed by one or more processors, cause the processors to perform operations comprising:
registering a plurality of input endpoint devices and a plurality of controlled endpoint devices, wherein each of the plurality of input endpoint devices has a respective audio input interface and is configured to transmit a captured voice input to the computing system over a network, and wherein each of the plurality of controlled endpoint devices is configured to perform a requested task in accordance with a corresponding instruction received from the computing system over the network; and
dynamically selecting a respective input endpoint device and a respective controlled device for each of a plurality of voice-based requests from a user to the computing system, including:
at a first point in time, acquiring respective instances of a first voice input from a first set of two or more input endpoint devices among the plurality of input endpoint devices;
obtaining a representative copy of the first voice input based on the respective instances of the first voice input that have been acquired from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a first actionable intent based on the representative copy of the first voice input; and
dispatching a first encoded instruction to a first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the first actionable intent.

### Claim 16 (depends on 15)

The computer-readable storage medium of claim 15, wherein the operations include:
registering a plurality of output endpoint devices, wherein each of the plurality of output endpoint devices has a respective audio output interface that is configured to output audio content received from the computing system over the network; and
dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, including:
dispatching a first audio output corresponding to the first actionable intent to a first output endpoint device selected from the plurality of output endpoint devices over the network, wherein the first output endpoint device outputs the first audio output.

### Claim 17 (depends on 16)

The computer-readable storage medium of claim 16, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
at a second point in time, acquiring respective instances of a second voice input from a second set of two or more input endpoint devices among the plurality of input endpoint devices, wherein the second set of two or more input endpoint devices are distinct from the first set of two or more input endpoint devices;
obtaining a representative copy of the second voice input based on the respective instances of the second voice input that have been acquired from the second set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a second actionable intent based on the representative copy of the second voice input; and
dispatching a second encoded instruction to the first controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the second actionable intent.

### Claim 18 (depends on 17)

The computer-readable storage medium of claim 17, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
dispatching a second audio output corresponding to the second actionable intent to a second output endpoint device selected from the plurality of output endpoint devices over the network, wherein the second output endpoint device is distinct from the first output endpoint device, and wherein the second output endpoint device outputs the first audio output.

### Claim 19 (depends on 16)

The computer-readable storage medium of claim 16, wherein dynamically selecting a respective output endpoint device for each of the plurality of voice-based requests from the user to the computing system, further includes:
at a third point in time, acquiring respective instances of a third voice input from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
obtaining a representative copy of the third voice input based on the respective instances of the third voice input that have been acquired from the first set of two or more input endpoint devices among the plurality of input endpoint devices;
determining a third actionable intent based on the representative copy of the third voice input; and
dispatching a third encoded instruction to a second controlled endpoint device selected from the plurality of controlled endpoint devices in accordance with the third actionable intent, wherein the second controlled endpoint device is distinct from the first controlled endpoint device.

### Claim 20 (depends on 15)

The computer-readable storage medium of claim 15, wherein the plurality of input endpoint devices and the plurality of controlled endpoint devices include devices located in a respective smart home environment.
