US 11,636,848 B2 · Token-wise training for attention based end-to-end speech recognition

Abstract

A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, includes performing cross-entropy training of a model, based on one or more input features of a speech signal, determining a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed, and determining a loss of the first wrong token at the time, based on the determined posterior probability vector. The method further includes determining a total loss of a training set of the model of which the cross-entropy training is performed, based on the determined loss of the first wrong token, and updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

Claims17

Dependent claims

A method of attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the method comprising:

performing cross-entropy training of a model, based on one or more input features of a speech signal;
determining a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed;
determining a loss of the first wrong token at the time, based on the determined posterior probability vector;
determining a total loss of a training set of the model of which the cross-entropy training is performed, based on
L
⁡
(
θ
)
TWT
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
ω
,
r
t
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, tω denotes the time, ytω denotes the first wrong token at the time, rtω denotes a reference token at the time, and lθ(ytω, rtω) denotes the loss of the first wrong token;
and
updating the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

35 more elements

An apparatus for attention-based end-to-end (A-E2E) automatic speech recognition (ASR) training, the apparatus comprising:

at least one memory configured to store program code;
and
at least one processor configured to read the program code and operate as instructed by the program code, the program code including:
performing code configured to cause the at least one processor to perform cross-entropy training of a model, based on one or more input features of a speech signal;
first determining code configured to cause the at least one processor to determine a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed;
second determining code configured to cause the at least one processor to determine a loss of the first wrong token at the time, based on the determined posterior probability vector;
third determining code configured to cause the at least one processor to determine a total loss of a training set of the model of which the cross-entropy training is performed, based on
L
⁡
(
θ
)
TWT
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
ω
,
r
t
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, tω denotes the time, ytω denotes the first wrong token at the time, rtω denotes a reference token at the time, and lθ(ytω, rtω) denotes the loss of the first wrong token;
and
updating code configured to cause the at least one processor to update the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

38 more elements

A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a device, cause the at least one processor to:

perform cross-entropy training of a model, based on one or more input features of a speech signal;
determine a posterior probability vector at a time of a first wrong token among one or more output tokens of the model of which the cross-entropy training is performed;
determine a loss of the first wrong token at the time, based on the determined posterior probability vector;
determine a total loss of a training set of the model of which the cross-entropy training is performed, based on
L
⁡
(
θ
)
TWT
=
∑
(
y
,
r
)
∈
(
Y
,
R
)
l
θ
(
y
t
ω
,
r
t
ω
)
,
where L(θ) denotes the total loss of the training set, (Y,R) denotes hypothesis-reference pairs in the training set, tω denotes the time, ytω denotes the first wrong token at the time, rtω denotes a reference token at the time, and Lθ(ytω, rtω) denotes the loss of the first wrong token and
update the model of which the cross-entropy training is performed, based on the determined total loss of the training set.

34 more elements

Dependencies are inferred from the claim text.

Token-wise training for attention based end-to-end speech recognition

Life of this patent

Token-wise training for attention based end-to-end speech recognition

Life of this patent

Abstract

Claims17