# US 2020/0265831 A1 · Large margin training for attention-based end-to-end speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0265831 A1
- Document ID: US20200265831A1
- Status: Published application
- Kind: A1
- Application number: 16/276,081
- Filed: 2019-02-14
- Published: 2020-08-20
- Earliest claimed: 2019-02-14
- Assignees: Tencent America LLC
- Applicants: Tencent America LLC
- Inventors: Peidong WANG; Jia CUI; Chao WENG; Dong YU
- Web page: https://patentgrep.com/patent/US20200265831A1

## CPC

- G10L 15/063: {Training}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 15/16: using artificial neural networks
- G10L 15/30: Distributed recognition, e.g. in client-server systems, for mobile phones or network applications
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of attention-based end-to-end (E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, performing beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses, and determining a one-best hypothesis among the generated n-best hypotheses list. The method further includes determining a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized, and performing backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.

## Claims

### Claim 1

A method of attention-based end-to-end (E2E) automatic speech recognition (ASR) training, the method comprising:
performing cross-entropy training of a model, based on one or more input features of a speech signal;
performing beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses;
determining a one-best hypothesis among the generated n-best hypotheses list;
determining a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized; and
performing backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.

### Claim 2 (depends on 1)

The method of claim 1, wherein the output hypotheses included in the generated n-best hypotheses list have highest posteriors among output sequences of the model of which the cross-entropy training is performed.

### Claim 3 (depends on 1)

The method of claim 1, wherein the loss function is represented as follows:
L

(
θ
)

∑
(
χ
,
s
)
∈
D

[
l

(
s
^
b
,

s
)
-
(
log


p
θ

(
s

χ
)
-
log


p
θ

(
s
^
b

χ
)
)
]
4
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝb denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, and θ denotes model parameters.

### Claim 4 (depends on 1)

The method of claim 1, wherein the loss function is represented as follows:
L

(
θ
)
=
∑
(
χ
,
s
)
∈
D

[
l

(
s
^
b
,
s
)
-
(
score
θ

(
s

χ
)
-
score
θ

(
s
^
b

χ
)
)
]
4
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝb denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, θ denotes model parameters, and score denotes a log posterior.

### Claim 5 (depends on 4)

The method of claim 4, wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L

(
θ
)
∂
log


p
θ

(
s
i
)
=
-
2


γ
+

δ

(
s
i
)
∂
L

(
θ
)
∂
log


p
θ

(
s
^
b
,
i
)
=

2


γ
+

δ

(
s
^
b
,
i
)
,
where δ(·) denotes a Kronecker delta function, and γ+ denotes [l(ŝb,s)−(scoreθ(s)−scoreθ(ŝb))].

### Claim 6 (depends on 4)

The method of claim 4, wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L

(
θ
)
∂
log


p
θ

(
s
i
)
=
-
2


γ
+

δ

(
s
i
)

δ

(
i
≥
w
)
∂
L

(
θ
)
∂
log


p
θ

(
s
^
b
,
i
)
=
2


γ
+

δ

(
s
^
b
,
i
)

δ

(
i
≥
w
)
,
where δ(·) denotes a Kronecker delta function, γ+ denotes [l(ŝb,s)−(scoreθ(s)−scoreθ(ŝb))], and ω denotes a first wrong token.

### Claim 7 (depends on 1)

The method of claim 1, further comprising, based on the backpropagation of the determined character-based gradient and the determined word-based gradient to the model being performed, performing again the beam searching of the model of which the cross-entropy training is performed, to regenerate the n-best hypotheses list.

### Claim 8

An apparatus for attention-based end-to-end (E2E) automatic speech recognition (ASR) training, the apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
first performing code configured to cause the at least one processor to perform cross-entropy training of a model, based on one or more input features of a speech signal;
second performing code configured to cause the at least one processor to perform beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses;
first determining code configured to cause the at least one processor to determine a one-best hypothesis among the generated n-best hypotheses list;
second and third determining code configured to cause the at least one processor to determine a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized; and
third performing code configured to cause the at least one processor to perform backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the output hypotheses included in the generated n-best hypotheses list have highest posteriors among output sequences of the model of which the cross-entropy training is performed.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the loss function is represented as follows:
L

(
θ
)

∑
(
χ
,
s
)
∈
D

[
l

(
s
^
b
,

s
)
-
(
log


p
θ

(
s

χ
)
-
log


p
θ

(
s
^
b

χ
)
)
]
4
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝb denotes the one-best hypothesis, l(ŝb, s) denotes the distance between the reference sequence and the determined one-best hypothesis, and θ denotes model parameters.

### Claim 11 (depends on 8)

The apparatus of claim 8, wherein the loss function is represented as follows:
L

(
θ
)
=
∑
(
χ
,
s
)
∈
D

[
l

(
s
^
b
,
s
)
-
(
score
θ

(
s

χ
)
-
score
θ

(
s
^
b

χ
)
)
]
4
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝb denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, θ denotes model parameters, and score denotes a log posterior.

### Claim 12 (depends on 11)

The apparatus of claim 11, wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L

(
θ
)
∂
log


p
θ

(
s
i
)
=
-
2


γ
+

δ

(
s
i
)
∂
L

(
θ
)
∂
log


p
θ

(
s
^
b
,
i
)
=

2


γ
+

δ

(
s
^
b
,
i
)
,
where δ(·) denotes a Kronecker delta function, and γ+ denotes [l(ŝb,s)−(scoreθ(s)−scoreθ(ŝb))].

### Claim 13 (depends on 11)

The apparatus of claim 11, wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L

(
θ
)
∂
log


p
θ

(
s
i
)
=
-
2


γ
+

δ

(
s
i
)

δ

(
i
≥
w
)
∂
L

(
θ
)
∂
log


p
θ

(
s
^
b
,
i
)
=
2


γ
+

δ

(
s
^
b
,
i
)

δ

(
i
≥
w
)
,
where δ(·) denotes a Kronecker delta function, γ+ denotes [l(ŝb,s)−(scoreθ(s)−scoreθ(ŝb))], and ω denotes a first wrong token.

### Claim 14 (depends on 8)

The apparatus of claim 8, wherein the second performing code is further configured to cause the at least one processor to, based on the backpropagation of the determined character-based gradient and the determined word-based gradient to the model being performed, perform again the beam searching of the model of which the cross-entropy training is performed, to regenerate the n-best hypotheses list.

### Claim 15

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the at least one processor to:
perform cross-entropy training of a model, based on one or more input features of a speech signal;
perform beam searching of the model of which the cross-entropy training is performed, to generate an n-best hypotheses list of output hypotheses;
determine a one-best hypothesis among the generated n-best hypotheses list;
determine a character-based gradient and a word-based gradient, based on the model of which the cross-entropy training is performed and a loss function in which a distance between a reference sequence and the determined one-best hypothesis is maximized; and
perform backpropagation of the determined character-based gradient and the determined word-based gradient to the model, to update the model.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the output hypotheses included in the generated n-best hypotheses list have highest posteriors among output sequences of the model of which the cross-entropy training is performed.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the loss function is represented as follows:
L

(
θ
)

∑
(
χ
,
s
)
∈
D

[
l

(
s
^
b
,

s
)
-
(
log


p
θ

(
s

χ
)
-
log


p
θ

(
s
^
b

χ
)
)
]
4
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝb denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, and θ denotes model parameters.

### Claim 18 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the loss function is represented as follows:
L

(
θ
)
=
∑
(
χ
,
s
)
∈
D

[
l

(
s
^
b
,
s
)
-
(
score
θ

(
s

χ
)
-
score
θ

(
s
^
b

χ
)
)
]
4
2
,
where (χ, s) denotes a sample in a training set D, χ denotes an input feature, s denotes the reference sample, ŝb denotes the one-best hypothesis, l(ŝ, s) denotes the distance between the reference sequence and the determined one-best hypothesis, θ denotes model parameters, and score denotes a log posterior.

### Claim 19 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L

(
θ
)
∂
log


p
θ

(
s
i
)
=
-
2


γ
+

δ

(
s
i
)
∂
L

(
θ
)
∂
log


p
θ

(
s
^
b
,
i
)
=

2


γ
+

δ

(
s
^
b
,
i
)
,
where δ(·) denotes a Kronecker delta function, and γ+ denotes [l(ŝb,s)−(scoreθ(s)−scoreθ(ŝb))].

### Claim 20 (depends on 18)

The non-transitory computer-readable medium of claim 18, wherein each of the character-based gradient and the word-based gradient is represented as follows:
∂
L

(
θ
)
∂
log


p
θ

(
s
i
)
=
-
2


γ
+

δ

(
s
i
)

δ

(
i
≥
w
)
∂
L

(
θ
)
∂
log


p
θ

(
s
^
b
,
i
)
=
2


γ
+

δ

(
s
^
b
,
i
)

δ

(
i
≥
w
)
,
where δ(·) denotes a Kronecker delta function, γ+ denotes [l(ŝb,s)−(scoreθ(s)−scoreθ(ŝb))], and ω denotes a first wrong token.
