# US 2021/0082406 A1 · Electronic device for analyzing meaning of speech, and operation method therefor

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0082406 A1
- Document ID: US20210082406A1
- Status: Published application
- Kind: A1
- Application number: 16/954,019
- Filed: 2018-11-07
- Published: 2021-03-18
- Earliest claimed: 2017-12-14
- Assignees: Samsung Electronics Co., Ltd.
- Applicants: Samsung Electronics Co., Ltd.
- Inventors: Jun Seong KIM
- Web page: https://patentgrep.com/patent/US20210082406A1

## CPC

- G10L 15/16: using artificial neural networks
- G06F 40/30: Semantic analysis
- G06F 40/35: Discourse or dialogue representation
- G06N 3/04: Architecture, e.g. interconnection topology
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G10L 15/063: {Training}
- G10L 15/1815: {Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning}
- G10L 15/183: using context dependencies, e.g. language models
- G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H)
- G06N 5/027: {Frames}
- G10L 15/1822: {Parsing for meaning understanding}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

An electronic device using an artificial neural network model including an attention mechanism, according to various embodiments, can comprise: a memory configured to store information including a plurality of recurrent neural network (RNN) layers; and at least one processor connected with the memory and configured to set, as a first key and a value, at least one first hidden representation acquired through at least one layer among the plurality of RNN layers, set, as a second key, at least one second hidden representation acquired through at least one second layer among the plurality of RNN layers, and acquire an attention included in an attention structure at least on the basis of data on the first key, data on the second key, or data on the value.

## Claims

### Claim 1

An electronic device using an artificial neural network model comprising an attention mechanism, the electronic device comprising:
a memory configured to store information comprising a plurality of recurrent neural network (RNN) layers; and
at least one processor connected with the memory,
wherein the at least one processor is configured to:
set, as a first key and a value, at least one first hidden representation obtained through at least one first layer from among the plurality of RNN layers;
set, as a second key, at least one second hidden representation obtained through at least one second layer from among the plurality of RNN layers; and
based at least on data on the first key, data on the second key, or data on the value, obtain an attention included in the attention mechanism.

### Claim 2 (depends on 1)

The electronic device of claim 1, wherein the at least one first layer is positioned at a lower stage than the at least one second layer in the artificial neural network, and
wherein the at least one processor is configured to obtain the at least one second hidden representation from the at least one first hidden representation through the at least one second layer.

### Claim 3 (depends on 2)

The electronic device of claim 2, wherein the at least one processor is configured to obtain, from the attention, data on a slot or data on an intent, through at least one third layer positioned at a higher stage than the at least one second layer from among the plurality of RNN layers.

### Claim 4 (depends on 1)

The electronic device of claim 1, wherein the at least one processor is configured to obtain data on a weight of each of the first key and the second key, based on the data on the first key and the data on the second key, and to obtain the attention based on a weighted sum of the data on the weight and the data on the value.

### Claim 5 (depends on 1)

The electronic device of claim 1, wherein the at least one processor is configured to obtain the attention based on the following equation:
L

A
i
=
∑
j
=
1
T
x

α
ij

h
j
1


α
ij
=
exp

(
e
i

j
)
Σ
k
=
1
T
x

exp

(
e
i

k
)


e
ij
=
a

(
s
i
-
1
,
h
j
1
,

h
j
2
)
where i indicates a step of decoding, j indicate an index of an input regarding the RNN, Tx indicates a length of an input sequence, si−1 indicates a decoder state at the i-th step of decoding, LAi indicates an attention regarding the i-th step of decoding, a indicates a neural network, αij indicates a weight regarding si−1, which is obtained by a, hj1 indicates the at least one first hidden presentation, and hj2 indicates the at least one second hidden representation.

### Claim 6 (depends on 5)

The electronic device of claim 5, wherein the at least one processor is configured to:
obtain data on a third key in which positional encoding is applied to the at least one first hidden representation;
obtain data on a fourth key in which the positional encoding is applied to the at least one second hidden representation; and
obtain the attention based at least on the third key and the fourth key.

### Claim 7 (depends on 1)

The electronic device of claim 1, wherein training a language model regarding the RNN, training a slot filling model regarding the RNN, and training an intent detection model regarding the RNN share at least one of at least one layer of the plurality of RNN layers, the at least one first hidden representation, or the at least one second hidden representation, and
wherein a cost for the RNN is obtained based at least on multiplication of data indicating a cost regarding the language model by a weight-decay term.

### Claim 8

A non-transitory computer-readable storage medium, which stores one or more programs for storing information comprising a plurality of recurrent neural network (RNN) layers, and for executing: setting, as a first key and a value, at least one first hidden representation obtained through at least one first layer from among the plurality of RNN layers; setting, as a second key, at least one second hidden representation obtained through at least one second layer from among the plurality of RNN layers; and, based at least on data on the first key, data on the second key, or data on the value, obtaining an attention included in an attention mechanism.

### Claim 9 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein the at least one first layer is positioned at a lower stage than the at least one second layer in the artificial neural network, and
wherein the non-transitory computer-readable storage medium further stores one or more programs for executing obtaining the at least one second hidden representation from the at least one first hidden representation through the at least one second layer.

### Claim 10 (depends on 9)

The non-transitory computer-readable storage medium of claim 9, further storing one or more programs for executing obtaining, from the attention, data on a slot or data on an intent, through at least one third layer positioned at a higher stage than the at least one second layer from among the plurality of RNN layers.

### Claim 11 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, further storing one or more programs for executing obtaining data on a weight of each of the first key and the second key, based on the data on the first key and the data on the second key, and obtaining the attention based on a weighted sum of the data on the weight and the data on the value.

### Claim 12 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, further storing one or more programs for executing obtaining the attention based on the following equation:
L

A
i
=
Σ
j
=
1
T
x

α
ij

h
j
1


α
ij
=
exp

(
e
i

j
)
Σ
k
=
1
T
x

exp

(
e
i

k
)
e
ij
=
a

(
s
i
-
1
,
h
j
1
,

h
j
2
)
where i indicates a step of decoding, j indicate an index of an input regarding the RNN, Tx indicates a length of an input sequence, si−1 indicates a decoder state at the i-th step of decoding, LAi indicates an attention regarding the i-th step of decoding, a indicates a neural network, αij indicates a weight regarding si−1, which is obtained by a, hj1 indicates the at least one first hidden presentation, and hj2 indicates the at least one second hidden representation.

### Claim 13 (depends on 12)

The non-transitory computer-readable storage medium of claim 12, further storing one or more programs for executing:
obtaining data on a third key in which positional encoding is applied to the at least one first hidden representation;
obtaining data on a fourth key in which the positional encoding is applied to the at least one second hidden representation; and
obtaining obtain the attention based at least on the third key and the fourth key.

### Claim 14 (depends on 8)

The non-transitory computer-readable storage medium of claim 8, wherein training a language model regarding the RNN, training a slot filling model regarding the RNN, and training an intent detection model regarding the RNN share at least one of at least one layer of the plurality of RNN layers, the at least one first hidden representation, or the at least one second hidden representation, and
wherein a cost for the RNN is obtained based at least on multiplication of data indicating a cost regarding the language model by a weight-decay term.

### Claim 15

An operation method of an electronic device using an artificial neural network model comprising an attention mechanism, the electronic device storing information comprising a plurality of recurrent neural network (RNN) layers, the method comprising:
setting, as a first key and a value, at least one first hidden representation obtained through at least one first layer from among the plurality of RNN layers;
setting, as a second key, at least one second hidden representation obtained through at least one second layer from among the plurality of RNN layers; and
based at least on data on the first key, data on the second key, or data on the value, obtaining an attention included in the attention mechanism.
