# US 2021/0264901 A1 · Token-wise training for attention based end-to-end speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2021/0264901 A1
- Document ID: US20210264901A1
- Status: Published application
- Kind: A1
- Application number: 17/316,856
- Filed: 2021-05-11
- Published: 2021-08-26
- Earliest claimed: 2019-02-14
- Assignees: Tencent America LLC
- Applicants: Tencent America LLC
- Inventors: Peidong WANG; Jia CUI; Chao WENG; Dong YU
- Web page: https://patentgrep.com/patent/US20210264901A1

## CPC

- G10L 15/063: {Training}
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/047: Probabilistic or stochastic networks
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 7/01: Probabilistic graphical models, e.g. probabilistic networks
- G06N 20/00: Machine learning
- G10L 15/14: using statistical models, e.g. Hidden Markov Models [HMMs] (G10L15/18 takes precedence)
- G10L 15/16: using artificial neural networks
- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, determining a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed, and determining a loss of the first wrong token at the time, based on the determined posterior probability vector. The method further includes determining a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token, and updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

## Claims

### Claim 1

A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the method comprising:
performing cross-entropy training of a model, based on one or more input features of a speech signal;
determining a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed;
determining a loss of the first wrong token at the time, based on the determined posterior probability vector;
determining a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token of each of hypothesis-reference pairs in the training set with respect to a reference token at the time; and
updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

### Claim 2 (depends on 1)

The method of claim 1, wherein the posterior probability vector at the time is determined as follows:
pt=Decoder(st−1 ∈ {rt−1, yt−1}, Henc),
where t denotes the time, pt denotes the posterior probability vector at the time t, Henc denotes the one or more features that are encoded, yt−1 denotes an output token at a previous time t−1, rt−1 denotes a reference token at the previous time t−1, and st−1 denotes a token randomly selected from {rt−1,yt−1}.

### Claim 3 (depends on 1)

The method of claim 1, wherein the total loss of the training set is determined as follows:
⁢
L
⁡
(
θ
)
TWT
=
∑
?
⁢
⁢
(
?
)
,
⁢
?
⁢
indicates text missing or illegible when filed
where L(θ) denotes the total loss of the training set, (Y,R) denotes the hypothesis-reference pairs in the training set, tω denotes the time, ytω denotes the first wrong token at the time, rtω denotes the reference token at the time, and lθ(ytω, rtω) denotes the loss of the first wrong token.

### Claim 4 (depends on 3)

The method of claim 3, wherein the loss of the first wrong token is determined as follows:
lθ(ytω, rtω)=−log ptω,rtω,
where ptω,rtω denotes a posterior probability of the reference token at the time.

### Claim 5 (depends on 3)

The method of claim 3, wherein the loss of the first wrong token is determined as follows:
lθ(ytω, rtω)=−log ptω,rtω+log ptω,ytω,
where ptω,rtω denotes a posterior probability of the reference token at the time, and ptω,ytω denotes a posterior probability of the first wrong token at the time.

### Claim 6 (depends on 1)

The method of claim 1, further comprising selecting a hypothesis with a longest correct prefix, from a plurality of hypotheses of the model of which the cross-entropy training is performed,
wherein the determining posterior probability vector at the time comprises determining the posterior probability vector at the time of the first wrong token included in the selected hypothesis.

### Claim 7 (depends on 6)

The method of claim 6, wherein the total loss of the training set is determined as follows:
⁢
L
⁡
(
θ
)
TWT
i
⁢
B
=
∑
?
⁢
⁢
l
θ
⁡
(
?
)
,
⁢
?
⁢
indicates text missing or illegible when filed
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, tjl,ω denotes the time, ytjl,ωjl denotes the first wrong token at the time, rtjl, ω denotes a reference token at the time, and lθ(ytjl,ωjl, rtjl,ω) denotes the loss of the first wrong token.

### Claim 8

An apparatus for attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the apparatus comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
performing code configured to cause the at least one processor to perform cross-entropy training of a model, based on one or more input features of a speech signal;
first determining code configured to cause the at least one processor to determine a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed;
second determining code configured to cause the at least one processor to determine a loss of the first wrong token at the time, based on the determined posterior probability vector;
third determining code configured to cause the at least one processor to determine a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token of each of hypothesis-reference pairs in the training set with respect to a reference token at the time; and
updating code configured to cause the at least one processor to update the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

### Claim 9 (depends on 8)

The apparatus of claim 8, wherein the posterior probability vector at the time is determined as follows:
pt=Decoder(st−1 ∈ {rt−1, yt−1}, Hend),
where t denotes the time, pt denotes the posterior probability vector at the time t, Henc denotes the one or more features that are encoded, yt−1 denotes an output token at a previous time t−1, rt−1 denotes a reference token at the previous time t−1, and st−1 denotes a token randomly selected from {rt−1,yt−1}.

### Claim 10 (depends on 8)

The apparatus of claim 8, wherein the total loss of the training set is determined as follows:
⁢
L
⁡
(
θ
)
TWT
=
∑
⁢
?
⁢
(
?
)
,
⁢
?
⁢
indicates text missing or illegible when filed
where L(θ) denotes the total loss of the training set, (Y,R) denotes the hypothesis-reference pairs in the training set, tω denotes the time, ytω denotes the first wrong token at the time, rtω denotes the reference token at the time, and lθ(ytω, rtω) denotes the loss of the first wrong token.

### Claim 11 (depends on 10)

The apparatus of claim 10, wherein the loss of the first wrong token is determined as follows:
ly(ytω, rtω)=−log ptω,rtω,
where ptω,rtω denotes a posterior probability of the reference token at the time.

### Claim 12 (depends on 10)

The apparatus of claim 10, wherein the loss of the first wrong token is determined as follows:
ly(ytω, rtω)=−log ptω,rtω+log ptω,ytω,
where ptω,rtω denotes a posterior probability of the reference token at the time, and ptω,ytω denotes a posterior probability of the first wrong token at the time.

### Claim 13 (depends on 8)

The apparatus of claim 8, further comprising selecting code configured to cause the at least one processor to select a hypothesis with a longest correct prefix, from a plurality of hypotheses of the model of which the cross-entropy training is performed,
wherein the first determining code is further configured to cause the at least one processor to determine the posterior probability vector at the time of the first wrong token included in the selected hypothesis.

### Claim 14 (depends on 13)

The apparatus of claim 13, wherein the total loss of the training set is determined as follows:
⁢
L
⁡
(
θ
)
TWT
i
⁢
B
=
∑
?
⁢
⁢
(
?
)
,
⁢
?
⁢
indicates text missing or illegible when filed
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, tjl,ω denotes the time, ytjl,ωjl denotes the first wrong token at the time, rtjl,ω denotes a reference token at the time, and lθ(ytjl,ωjl, rtjl,ω) denotes the loss of the first wrong token.

### Claim 15

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the at least one processor to:
perform cross-entropy training of a model, based on one or more input features of a speech signal;
determine a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed;
determine a loss of the first wrong token at the time, based on the determined posterior probability vector;
determine a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token of each of hypothesis-reference pairs in the training set with respect to a reference token at the time; and
update the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

### Claim 16 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the posterior probability vector at the time is determined as follows:
pt=Decoder(st−1 ∈ {rt−1, yt−1}, Henc),
where t denotes the time, pt denotes the posterior probability vector at the time t, Henc denotes the one or more features that are encoded, yt−1 denotes an output token at a previous time t−1, rt−1 denotes a reference token at the previous time t−1, and st−1 denotes a token randomly selected from {rt−1,yt−1}.

### Claim 17 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the total loss of the training set is determined as follows:
⁢
L
⁡
(
θ
)
TWT
=
∑
⁢
?
⁢
(
?
)
,
⁢
?
⁢
indicates text missing or illegible when filed
where L(θ) denotes the total loss of the training set, (Y,R) denotes the hypothesis-reference pairs in the training set, tω denotes the time, ytω denotes the first wrong token at the time, rtω denotes the reference token at the time, and lθ(ytω, rtω) denotes the loss of the first wrong token.

### Claim 18 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the loss of the first wrong token is determined as follows:
lθ(ytω, rtω)=−log ptω,rtω,
where ptω,rtω denotes a posterior probability of the reference token at the time.

### Claim 19 (depends on 17)

The non-transitory computer-readable medium of claim 17, wherein the loss of the first wrong token is determined as follows:
lθ(ytω, rtω)=−log ptω,rtω+log ptω,ytω,
where ptω,rtω denotes a posterior probability of the reference token at the time, and ptωytω denotes a posterior probability of the first wrong token at the time.

### Claim 20 (depends on 15)

The non-transitory computer-readable medium of claim 15, wherein the instructions further cause the at least one processor to:
select a hypothesis with a longest correct prefix, from a plurality of hypotheses of the model of which the cross-entropy training is performed; and
determine the posterior probability vector at the time of the first wrong token included in the selected hypothesis.
