# US 11,908,465 B2 · Electronic device and controlling method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,465 B2
- Document ID: US11908465B2
- Status: Granted
- Kind: B2
- Application number: 16/893,643
- Filed: 2020-06-05
- Published: 2020-09-24
- Granted: 2024-02-20
- Earliest claimed: 2016-11-03
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Young-chul Sohn; Gyu-tae Park; Ki-beom Lee; Jong-ryul Lee
- Web page: https://patentgrep.com/patent/US11908465B2

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06F 3/011: {Arrangements for interaction with the human body, e.g. for user immersion in virtual reality (blind teaching G09B21/00)}
- G06F 3/167: {Audio in a user interface, e.g. using voice commands for navigating, audio feedback}
- G10L 15/18: using natural language modelling
- G10L 17/22: Interactive procedures; Man-machine interfaces
- H04L 12/28: characterised by path configuration, e.g. LAN [Local Area Networks] or WAN [Wide Area Networks] (wireless communication networks H04W {; arrangements for dividing the transmission path H04W40/00})
- H04L 12/282: {based on user interaction within the home (receiver circuitry for displaying additional information being controlled by a remote control apparatus H04N21/42204)}
- H04N 21/4131: {home appliance, e.g. lighting, air conditioning system, metering devices (home automation data switching networks exchanging configuration information on appliance services H04L12/2807)}
- H04N 21/42203: {sound input device, e.g. microphone}
- H04N 21/4223: Cameras (H04N23/00 takes precedence)
- H04N 21/4394: {involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams (arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58)}
- H04N 21/44008: {involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream (arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59)}
- G06F 2203/0381: Multimodal input, i.e. interface arrangements enabling the user to issue commands by simultaneous use of input devices of different nature, e.g. voice plus gesture on digitizer
- G10L 2015/088: {Word spotting}
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M)
- H04N: PICTORIAL COMMUNICATION, e.g. TELEVISION

## Abstract

An approach for controlling method of an electronic device is provided. The approach acquires voice information and image information for setting an action to be executed according to a condition, the voice information and the image information being respectively generated from a voice and a behavior associated with the voice of a user. The approach determines an event to be detected according to the condition and a function to be executed according to the action when the event is detected, based on the acquired voice information and the acquired image information. The approach determines at least one detection resource to detect the determined event. In response to the at least one determined detection resource detecting at least one event satisfying the condition, the approach executes the function according to the action.

## Claims

### Claim 1

A controlling method of an electronic device, the method comprising:
acquiring voice information and image information for setting a condition and an action to be executed according to the condition, the voice information being obtained from a user voice corresponding to a single voice input, the image information being obtained via a camera;
identifying an event that is to be detected according to the condition and a function to be executed according to the action based on the voice information and the image information, and storing the event and the function;
identifying at least one external sensor, distinct from the camera, to detect the event;
based on the at least one external sensor being identified, transmitting control information requesting detection of the event to the at least one external sensor; and
based on receiving a result of detection of the event from the at least one external sensor, performing an operation for executing the function according to the action.

### Claim 2 (depends on 1)

The method of claim 1, wherein the identifying the at least one external sensor comprises:
retrieving pre-installed available resources; and
identifying at least one external sensor, among the retrieved pre-installed available resources, to detect the event using a detection function of the at least one external sensor.

### Claim 3 (depends on 1)

The method of claim 1, further comprising:
in response to the at least one external sensor being identified, transmitting control information requesting detection of the event to the at least one identified external sensor.

### Claim 4 (depends on 1)

The method of claim 1, further comprising:
retrieving pre-installed available resources; and
identifying at least one execution resource, among the retrieved pre-installed available resources, to execute the function according to the action using an execution function of the identified at least one execution resource.

### Claim 5 (depends on 1)

The method of claim 1, wherein the performing an operation comprises:
receiving a result of detection of the event from the detection resource; and
executing the function according the action based on the received detection result.

### Claim 6 (depends on 1)

The method of claim 1, further comprising providing, in response to there being no external sensor to detect the event or in response to the external sensor not being capable of detecting the event, a notification user interface (UI) notifying that execution of the action according to the condition is not possible.

### Claim 7 (depends on 1)

The method of claim 1, wherein the identifying the event to be detected comprises identifying the condition and the action according to an intent of a user by applying the voice information to a data recognition model generated using a learning algorithm.

### Claim 8 (depends on 7)

The method of claim 7, wherein the identifying the condition and the action according to the intent of the user further comprises:
providing a notification user interface (UI) for identifying the condition and the action to the user.

### Claim 9

An electronic device, comprising:
a memory;
communicator; and
a processor configured:
to acquire voice information and image information for setting a condition and an action to be executed according to the condition, the voice information being obtained from a user voice corresponding to a single voice input, the image information being obtained via a camera,
to identify an event that is to be detected according to the condition and a function to be executed according to the action, based on the voice information, and to store the event and the function,
to identify at least one external sensor, distinct from the camera, to detect the event,
based on the at least one external sensor being identified, transmit control information requesting detection of the event to the at least one external sensor, and
to perform, based on receiving a result of detection of the event from the at least one external sensor, an operation for executing the function according to the action.

### Claim 10 (depends on 9)

The electronic device of claim 9, wherein the processor is further configured:
to retrieve, in response to identifying the at least one external sensor, pre-installed available resources, and
to identify at least one external sensor, among the retrieved pre-installed available resources, to detect the event using a detection function of the at least one external sensor.

### Claim 11 (depends on 9)

The device of claim 9, wherein the electronic device further comprises a communicator configured to communicate with the at least one external sensor, and
wherein the processor is further configured to control, in response to the at least one external sensor being identified, the communicator to transmit control information requesting for detection of the event to the at least one identified external sensor.

### Claim 12 (depends on 9)

The device of claim 9, wherein the processor is further configured:
to retrieve pre-installed available resources, and
to identify at least one execution resource, among the retrieved pre-installed available resources, to execute the function according to the action using an execution function of the identified at least one execution resource.

### Claim 13 (depends on 9)

The device of claim 9, wherein the processor is further configured:
to receive, in response to executing the function according to the action, a result of detection of the event from the external sensor, and
to execute the function according to the action based on the received detection result.

### Claim 14 (depends on 9)

The device of claim 9, wherein the electronic device further comprises a display configured to display a user interface (UI), and
wherein the processor is further configured to control, in response to there being no external sensor to detect the event or in response to the external sensor not being capable of detecting the event, the display to display a notification UI informing that execution of the action according to the action is not possible.

### Claim 15 (depends on 9)

The device of claim 9, wherein the processor is further configured to:
identify, in response to identifying a function to be executed according to an event to be detected and the action according to the condition based on the voice information, the condition and action according to an intent of a user by applying the voice information to a data recognition model generated using a learning algorithm, and
identify an event to be detected according to the condition and a function to be executed according to the action.

### Claim 16 (depends on 15)

The device of claim 15, wherein the electronic device further comprises a display configured to display a user interface (UI), and
wherein the processor is further configured to control, in response to identifying the condition and the action according to the intent of the user, the display to display a notification UI for identifying the condition and the action to the user.