# US 2021/0174810 A1 · Electronic device performing operation based on user speech in multi device environment and operating method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0174810 A1
- Document ID: US20210174810A1
- Status: Published application
- Kind: A1
- Application number: 17/114,381
- Filed: 2020-12-07
- Published: 2021-06-10
- Earliest claimed: 2019-12-05
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Taegu KIM; Hansin KOH; Jooyeon KIM; Hyeonjae BAK; Hyunjin PARK; Kyounggu WOO
- Web page: https://patentgrep.com/patent/US20210174810A1

## CPC

- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

According to various embodiments, an electronic device performing an operation corresponding to an utterance of a user in a multi-device environment includes a communication module, a processor, and a memory. The processor operates a first electronic device for receiving and a second electronic device for executing in association with each other. The memory stores instructions that, when being executed, cause the processor to: determine whether a first utterance received at the first electronic device should be performed at the second electronic device; when the first utterance should be performed at the second electronic device, form a connection binding with the first electronic device and the second electronic device by using one or more channels through the communication module. When an operation corresponding to the first utterance is completed at the second electronic device, the processor terminates the connection binding with the first electronic device and the second electronic device.

## Claims

### Claim 1

An electronic device performing an operation corresponding to an utterance of a user in a multi-device environment, the electronic device comprising:
a communication module;
a processor operatively connected with the communication module and configured to operate a first electronic device for receiving and a second electronic device for executing in association with each other; and
a memory operatively connected with the processor,
wherein the memory stores instructions that when executed, cause the processor to:
determine whether a first utterance received at the first electronic device should be performed at the second electronic device;
in response to determining that the first utterance should be performed at the second electronic device, form a connection binding with the first electronic device and the second electronic device by using one or more channels through the communication module; and
in response to determining that an operation corresponding to the first utterance is completed at the second electronic device, terminate the connection binding with the first electronic device and the second electronic device.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
receive first device information on the first electronic device and second device information on the second electronic device; and
based on the first device information and the second device information, connect the first electronic device and the second electronic device.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the instructions cause the processor to:
after the operation corresponding to the first utterance is completed, transmit a response resulting from processing of the first utterance to the first electronic device in the form of a user interface (UI) expressible by the first electronic device, based on the first device information.

### Claim 4 (depends on 3)

The electronic device of claim 3, wherein the instructions cause the processor to abstain from transmitting a response that is not the UI expressible by the first electronic device to the first electronic device.

### Claim 5 (depends on 2)

The electronic device of claim 2, wherein the instructions cause the processor to:
after the operation corresponding to the first utterance is completed, transmit a response resulting from processing of the first utterance to the second electronic device in the form of a user interface (UI) expressible by the second electronic device, based on the second device information.

### Claim 6 (depends on 3)

The electronic device of claim 3, wherein the instructions cause the processor to abstain from transmitting a response that is not UI expressible by the second electronic device to the second electronic device.

### Claim 7 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
after the operation corresponding to the first utterance is completed, correct a response content of the second electronic device resulting from processing of the first utterance, and transmit the corrected response content to the first electronic device.

### Claim 8 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
in response to the first electronic device and the second electronic device being connected, receive a first response format from the first electronic device; and
transmit a response resulting from processing of the first utterance to the first electronic device, based on the first response format.

### Claim 9 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
in response to the first electronic device and the second electronic device being connected, receive a second response format from the second electronic device; and
transmit a response resulting from processing of the first utterance to the second electronic device, based on the second response format.

### Claim 10 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
count a first connection maintaining time from a time that a first channel for connecting with the first electronic device is generated, and maintain or terminate the connection with the first electronic device based on the first connection maintaining time; and
count a second connection maintaining time from a time that a second channel for connecting with the second electronic device is generated, and maintain or terminate the connection with the second electronic device based on the second connection maintaining time.

### Claim 11 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
count a follow-up utterance waiting time from a time that the first utterance is received from the first electronic device; and
maintain or terminate the connection with the first electronic device, based on the follow-up utterance waiting time.

### Claim 12 (depends on 10)

The electronic device of claim 10, wherein the instructions cause the processor to:
set an additional information input waiting time for waiting until additional information is received from the user after the additional information is requested from the user; and
in response to the additional information being received within the additional information input waiting time or the additional information not being received for the additional information input waiting time, terminate the connection with the first electronic device and the second electronic device.

### Claim 13 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
in response to a new second utterance that should be performed at the second electronic device being received at the first electronic device after the connection with the first electronic device and the second electronic device is terminated, reconnect the first electronic device and the second electronic device.

### Claim 14 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
in a state that the connection with the second electronic device based on the first utterance is maintained,
in response to a new second utterance being received at a third electronic device for receiving, generate a connection with the third electronic device for receiving; and
connect the second electronic device and the third electronic device.

### Claim 15 (depends on 1)

The electronic device of claim 1, wherein the instructions cause the processor to:
in a state that the connection with the first electronic device based on the first utterance is maintained,
in response to a new second utterance that is determined to be performed at a fourth electronic device for executing is received at the first electronic device, generate a connection with the fourth electronic device for executing; and
connect the first electronic device and the fourth electronic device.

### Claim 16

An operating method of an electronic device that operates a first electronic device for receiving and a second electronic device for executing in association with each other, the method comprising:
determining whether a first utterance received at the first electronic device should be performed at the second electronic device;
in response to the first utterance being determined to be performed at the second electronic device, forming a connection binding with the first electronic device and the second electronic device by using one or more channels; and
in response to an operation corresponding to the first utterance being completed at the second electronic device, terminating the connection binding with the first electronic device and the second electronic device.

### Claim 17 (depends on 16)

The method of claim 16, further comprising:
receiving first device information on the first electronic device and second device information on the second electronic device; and
based on the first device information and the second device information, connecting the first electronic device and the second electronic device.

### Claim 18 (depends on 17)

The method of claim 17, further comprising:
after the operation corresponding to the first utterance is completed,
transmitting a response resulting from processing of the first utterance to the first electronic device in the form of a user interface (UI) expressible by the first electronic device, based on the first device information; and
transmitting a response resulting from processing of the first utterance to the second electronic device in the form of the UI expressible by the second electronic device, based on the second device information.

### Claim 19 (depends on 18)

The method of claim 18, wherein a response that is not the UI expressible by the first electronic device is not transmitted to the first electronic device, and
a response that is not the UI expressible by the second electronic device is not transmitted to the second electronic device.

### Claim 20 (depends on 16)

The method of claim 16, further comprising
after the operation corresponding to the first utterance is completed, correcting a response content of the second electronic device resulting from processing of the first utterance, and transmitting the corrected response content to the first electronic device.

### Claim 21

An electronic device performing an operation corresponding to an utterance of a user in a multi-device environment, the electronic device comprising:
a communication module;
a processor operatively connected with the communication module and configured to operate a first electronic device for receiving and a second electronic device for executing in association with each other; and
a memory operatively connected with the processor,
wherein the memory stores instructions that when executed, cause the processor to:
generate a first channel with the first electronic device that receives the utterance;
receive information of the utterance through the first channel;
determine whether an action corresponding to the utterance should be executed at the second electronic device, based on the information of the utterance;
generate a second channel with the second electronic device;
request the second electronic device to execute the action corresponding to the utterance through the second channel;
receive a result of executing the action corresponding to the utterance from the second electronic device; and
transmit the result of executing the action corresponding to the utterance to the first electronic device.

### Claim 22 (depends on 21)

The electronic device of claim 21, wherein the processor is further configured to:
maintain the first channel for a predetermined first connection maintaining time, and
maintain the second channel for a predetermined second connection maintaining time.

### Claim 23 (depends on 22)

The electronic device of claim 22, wherein, in response to any one of the first connection maintaining time and the second connection maintaining time is ended, the processor is configured to:
disable the first channel and the second channel.

### Claim 24 (depends on 22)

The electronic device of claim 22, wherein, in a state that the first connection maintaining time and the second connection maintaining time are effective, in response to an operation of continuing connection maintenance is executed at any one of the first electronic device and the second electronic device, the processor is further configured to:
reset the first connection maintaining time and the second connection maintaining time.

### Claim 25 (depends on 21)

The electronic device of claim 21, wherein, in a state that a first connection maintaining time and a second connection maintaining time are effective, in response to a connection termination (unlink) request being received from the first electronic device, terminate the connection with the second electronic device.

### Claim 26

An electronic device for receiving an utterance of a user in a multi-device environment, the electronic device comprising:
a communication module;
a processor operatively connected with the communication module and configured to operate a first electronic device for receiving and a second electronic device for executing in association with each other; and
a memory operatively connected with the processor,
wherein the memory stores instructions that when executed, cause the processor to:
in response to the utterance being received,
generate at least one channel and connect with the second electronic device;
display the connection with the second electronic device through a user interface (UI);
request the second electronic device to execute an action corresponding to the utterance through the at least one channel;
receive a result of executing the action corresponding to the utterance from the second electronic device; and
display the result of executing the action corresponding to the utterance through the UI.

### Claim 27 (depends on 26)

The electronic device of claim 26, wherein the processor is further configured to:
maintain the at least one channel for a predetermined connection maintaining time; and
disable the at least one channel in response to the connection maintaining time being ended.

### Claim 28 (depends on 27)

The electronic device of claim 27, wherein, in a state that the connection maintaining time is effective, in response to an operation of continuing connection maintenance being executed at the first electronic device, reset the connection maintaining time.

### Claim 29 (depends on 26)

The electronic device of claim 26, wherein the processor is further configured to:
in response to a new utterance being received in a state that a connection maintaining time is effective, determine a device to execute an action corresponding to the new utterance;
in response to determining that the new utterance should be executed at another device, display a connection termination (unlink) menu through the UI; and
terminate the connection with the second electronic device in response to the connection termination being selected by a user.

### Claim 30 (depends on 29)

The electronic device of claim 29, wherein the processor is further configured to:
connect the another device and the first electronic device; and
request the another device to execute an action corresponding to the new utterance.

### Claim 31 (depends on 26)

The electronic device of claim 26, wherein the processor is further configured to:
in response to a new utterance being received in a state that a connection maintaining time is effective, determine a device to execute the new utterance;
in response to determining that the new utterance should be executed at the first electronic device, return to a home screen of the first electronic device; and
execute an action corresponding to the new utterance.
