# US 12,763,069 B2 · Electronic apparatus and controlling method thereof

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,763,069 B2
- Document ID: US12763069B2
- Status: Granted
- Kind: B2
- Application number: 18/518,138
- Filed: 2023-11-22
- Published: 2024-08-29
- Granted: 2026-10-13
- Earliest claimed: 2023-02-23
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jaechul Yang; Munjo Kim; Yeonho Lee
- Web page: https://patentgrep.com/patent/US12763069B2

## CPC

- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G06F 16/632: Query formulation
- G06F 40/30: Semantic analysis
- G10L 15/08: Speech classification or search
- G10L 15/1822: {Parsing for meaning understanding}
- G10L 15/183: using context dependencies, e.g. language models
- G06F 40/35: Discourse or dialogue representation
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 2015/223: {Execution procedure of a spoken command}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

An electronic apparatus includes a microphone; a speaker; a memory storing at least one instruction; and one or more processors operatively coupled to the memory and the speaker, wherein the one or more processors are configured to execute the at least one instruction to: based on identifying that a first voice sensed through the microphone corresponds to a wake-up voice, convert a state of the electronic apparatus from a standby state to a wake-up state, based on a second voice being sensed while the state of the electronic apparatus is the wake-up state, identify a first user query included in first voice data obtained based on the second voice, perform a first operation corresponding to the first user query, identify a predetermined query corresponding to the first user query, obtain a query list including at least one query based on the predetermined query and user context information, identify a second user query included in second voice data obtained based on a third voice sensed through the microphone, and based on a first semantic similarity between the second user query and the at least one query of the query list being greater than or equal to a predetermined value, perform a second operation corresponding to the second user query and maintain the state of the electronic apparatus as the wake-up state.

## Claims

### Claim 1

An electronic apparatus comprising:
a microphone;
a speaker;
a memory storing at least one instruction; and
one or more processors;
wherein the one or more processors are configured to execute the at least one instruction to:
based on identifying that a first voice sensed through the microphone corresponds to a wake-up voice, convert a state of the electronic apparatus from a standby state to a wake-up state,
based on a second voice being sensed while the state of the electronic apparatus is the wake-up state, identify a first user query included in first voice data obtained based on the second voice,
perform a first operation corresponding to the first user query,
identify a predetermined query corresponding to the first user query,
obtain a query list comprising at least one query based on the predetermined query and user context information,
identify a relevance to the first user query for each of the at least one query of the query list based on the predetermined query and the user context information,
identify, from among the query list, a corresponding query having a maximum relevance to the first user query,
identify a second user query included in second voice data obtained based on a third voice sensed through the microphone,
identify a first semantic similarity between the second user query and the corresponding query having the maximum relevance,
based on the first semantic similarity being greater than or equal to a predetermined value, perform a second operation corresponding to the second user query and maintain the state of the electronic apparatus as the wake-up state,
identify a second semantic similarity between the second user query and the corresponding query,
based on the second semantic similarity being greater than or equal to the predetermined value, perform the second operation corresponding to the second user query and maintain the state of the electronic apparatus as the wake-up state, and
based on the second semantic similarity being less than the predetermined value, convert the state of the electronic apparatus from the wake-up state to the standby state and prevent performing the second operation corresponding to the second user query.

### Claim 2 (depends on 1)

The electronic apparatus of claim 1, wherein the one or more processors are further configured to execute the at least one instruction to:
based on the first semantic similarity being less than the predetermined value, change the state of the electronic apparatus from the wake-up state to the standby state without performing the second operation corresponding to the second user query.

### Claim 3 (depends on 1)

The electronic apparatus of claim 1, wherein the one or more processors are further configured to execute the at least one instruction to:
identify a domain of the first user query, and
identify the predetermined query based on the domain.

### Claim 4 (depends on 1)

The electronic apparatus of claim 1, wherein the user context information comprises at least one of a query history of a user, a query response history of the user, a location of the user, a current time, an ambient temperature, a current state of the electronic apparatus, or a use history of the electronic apparatus by the user.

### Claim 5 (depends on 1)

The electronic apparatus of claim 1, wherein the user context information comprises at least one of a query history of a user, a query response history of the user, a location of the user, a current time, an ambient temperature, a current state of the electronic apparatus, or a use history of the electronic apparatus by the user, and
wherein the one or more processors are further configured to execute the at least one instruction to:
identify a domain of the first user query,
identify the predetermined query based on the domain,
allocate a predetermined weight corresponding to the domain to the user context information, and
obtain the query list comprising the at least one query based on the predetermined query, the user context information, and the predetermined weight allocated to the user context information.

### Claim 6 (depends on 1)

The electronic apparatus of claim 1, wherein the one or more processors are further configured to execute the at least one instruction to:
exclude, from the query list, an unperformable query corresponding to an operation that is not currently performable by the electronic apparatus.

### Claim 7 (depends on 1)

The electronic apparatus of claim 1, wherein the one or more processors are further configured to execute the at least one instruction to:
identify a domain of the second user query,
based on the domain requiring user confirmation, control the speaker to output a voice requesting confirmation on whether the second operation corresponding to the second user query is to be performed,
based on content approving to perform the second operation, perform the second operation corresponding to the second user query and maintain the state of the electronic apparatus as the wake-up state, the content being included in third voice data obtained based on a fourth voice sensed through the microphone, and
based on the content not approving to perform the second operation, change the state of the electronic apparatus from the wake-up state to the standby state and prevent performing the second operation corresponding to the second user query.

### Claim 8 (depends on 1)

The electronic apparatus of claim 1, wherein the one or more processors are further configured to execute the at least one instruction to:
identify an elapsed time from a first time point when the second voice is sensed through the microphone to a second time point when the third voice is sensed through the microphone, and
based on the elapsed time being less than or equal to a predetermined time, identify the second user query from the second voice data obtained based on the third voice.

### Claim 9 (depends on 1)

The electronic apparatus of claim 1, wherein the one or more processors are further configured to execute the at least one instruction to:
obtain the query list comprising the at least one query based on a first vector value output by a query prediction model based on the predetermined query and the user context information, and
wherein the query prediction model is trained based on a second vector value generated by the query prediction model based on providing a third user query and the user context information to the query prediction model.

### Claim 10

A controlling method of an electronic apparatus, the controlling method comprising:
based on identifying that a first voice sensed corresponds to a wake-up voice, converting a state of the electronic apparatus from a standby state to a wake-up state;
based on a second voice being sensed while the state of the electronic apparatus is the wake-up state, identifying a first user query included in first voice data obtained based on the second voice sensed through a microphone of the electronic apparatus;
performing a first operation corresponding to the first user query;
identifying a predetermined query corresponding to the first user query;
obtaining a query list comprising at least one query based on the predetermined query and user context information;
identifying a relevance to the first user query for each of the at least one query of the query list based on the predetermined query and the user context information;
identifying, from among the query list, a corresponding query having a maximum relevance to the first user query;
identifying a second user query included in second voice data obtained based on a third voice sensed through the microphone;
identifying a first semantic similarity between the second user query and the corresponding query having the maximum relevance;
based on the first semantic similarity being greater than or equal to a predetermined value, performing a second operation corresponding to the second user query and maintaining the state of the electronic apparatus as the wake-up state;
identifying a second semantic similarity between the second user query and the corresponding query;
based on the second semantic similarity being greater than or equal to the predetermined value, performing the second operation corresponding to the second user query and maintaining the state of the electronic apparatus as the wake-up state; and
based on the second semantic similarity being less than the predetermined value, changing the state of the electronic apparatus from the wake-up state to the standby state and preventing the performing of the second operation corresponding to the second user query.

### Claim 11 (depends on 10)

The controlling method of claim 10, further comprising:
based on the first semantic similarity being less than the predetermined value, changing the state of the electronic apparatus from the wake-up state to the standby state without performing of the second operation corresponding to the second user query.

### Claim 12 (depends on 10)

The controlling method of claim 10, wherein the identifying the predetermined query comprises:
determining a domain of the first user query; and
identifying the predetermined query based on the domain.

### Claim 13 (depends on 10)

The controlling method of claim 10, wherein the user context information comprises at least one of a query history of a user, a query response history of the user, a location of the user, a current time, an ambient temperature, a current state of the electronic apparatus, or a use history of the electronic apparatus by the user.

### Claim 14 (depends on 10)

The controlling method of claim 10, wherein the user context information comprises at least one of a query history of a user, a query response history of the user, a location of the user, a current time, an ambient temperature, a current state of the electronic apparatus, or a use history of the electronic apparatus by the user, and
wherein the controlling method further comprises:
determining a domain of the first user query;
identifying the predetermined query based on the domain; and
allocating a predetermined weight corresponding to the domain to the user context information, and
wherein the obtaining the query list comprises obtaining the query list comprising the at least one query based on the predetermined query, the user context information, and the predetermined weight allocated to the user context information.

### Claim 15 (depends on 10)

The controlling method of claim 10, further comprising:
determining a domain of the second user query;
based on the domain requiring user confirmation, outputting a voice requesting confirmation on whether the second operation corresponding to the second user query is to be performed;
based on content approving to perform the second operation, performing the second operation corresponding to the second user query and maintaining the state of the electronic apparatus as the wake-up state, the content being included in third voice data obtained based on a fourth voice sensed through the microphone; and
based on the content not approving to perform the second operation, changing the state of the electronic apparatus from the wake-up state to the standby state and preventing the performing of the second operation corresponding to the second user query.

### Claim 16 (depends on 10)

The controlling method of claim 10, further comprising:
identifying an elapsed time from a first time point when the second voice is sensed through the microphone to a second time point when the third voice is sensed through the microphone; and
based on the elapsed time being less than or equal to a predetermined time, extracting the second user query from the second voice data obtained based on the third voice.

### Claim 17 (depends on 10)

The controlling method of claim 10, wherein the obtaining the query list comprises obtaining the query list comprising the at least one query based on a first vector value output by a query prediction model based on the predetermined query and the user context information, and
wherein the query prediction model is trained based on a second vector value generated by the query prediction model based on providing a third user query and the user context information to the query prediction model.

### Claim 18

A non-transitory computer readable recording medium including a program executes a controlling method of an electronic apparatus, the controlling method comprising:
based on identifying that a first voice sensed corresponds to a wake-up voice, converting a state of the electronic apparatus from a standby state to a wake-up state;
based on a second voice being sensed while the state of the electronic apparatus is the wake-up state, identifying a first user query included in first voice data obtained based on the second voice sensed through a microphone of the electronic apparatus;
performing a first operation corresponding to the first user query;
identifying a predetermined query corresponding to the first user query;
obtaining a query list comprising at least one query based on the predetermined query and user context information;
identifying a relevance to the first user query for each of the at least one query of the query list based on the predetermined query and the user context information;
identifying, from among the query list, a corresponding query having a maximum relevance to the first user query;
identifying a second user query included in second voice data obtained based on a third voice sensed through the microphone;
identifying a semantic similarity between the second user query and the corresponding query having the maximum relevance;
based on the semantic similarity between the second user query and the at least one query of the query list being greater than or equal to a predetermined value, performing a second operation corresponding to the second user query and maintaining the state of the electronic apparatus as the wake-up state;
identifying a second semantic similarity between the second user query and the corresponding query;
based on the second semantic similarity being greater than or equal to the predetermined value, performing the second operation corresponding to the second user query and maintain the state of the electronic apparatus as the wake-up state; and
based on the second semantic similarity being less than the predetermined value, converting the state of the electronic apparatus from the wake-up state to the standby state and prevent performing the second operation corresponding to the second user query.