Attention-based joint acoustic and text on-device end-to-end model
AssigneeGoogle Inc.
InventorsTara N. Sainath, Ruoming Pang, Ron Weiss, Yanzhang He, Chung-cheng Chiu, Trevor Strohman
Life of this patent
Estimating the term…
Abstract
A method includes receiving a training example for a listen-attend-spell (LAS) decoder of a two-pass streaming neural network model and determining whether the training example corresponds to a supervised audio-text pair or an unpaired text sequence. When the training example corresponds to an unpaired text sequence, the method also includes determining a cross entropy loss based on a log probability associated with a context vector of the training example. The method also includes updating the LAS decoder and the context vector based on the determined cross entropy loss.
Claims20
A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
- receiving a sequence of acoustic frames characterizing an utterance;
- encoding, using a shared encoder, each acoustic frame in the sequence of acoustic frames to generate a corresponding encoded acoustic frame;
- generating, using a first-pass decoder, based on the encoded acoustic frames, streamed hypotheses;
- at each output step of a plurality of output steps for a second-pass decoder:
- determining, using a same attention mechanism, both:
- an acoustic context vector that summarizes the encoded acoustic frames;
- and
- a linguistic context vector based on a sequence of decoded labels previously output by the second-pass decoder;
- determining, using the acoustic context vector, as output from the second-pass decoder, an acoustic-based probability distribution over possible output labels;
- determining, using the linguistic context vector, as output from the second-pass decoder, a text-based probability distribution over possible output labels;
- and
- interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels, wherein the possible output labels in the acoustic-based probability distribution and the possible output labels in the text-based probability distribution comprise a same set subword units;
- and
- determining a transcription of the utterance based on the interpolating of the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels at each of the plurality of output steps.
A system comprising:
- data processing hardware;
- and
- memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
- receiving a sequence of acoustic frames characterizing an utterance;
- encoding, using a shared encoder, each acoustic frame in the sequence of acoustic frames to generate a corresponding encoded acoustic frame;
- generating, using a first-pass decoder, based on the encoded acoustic frames, streamed hypotheses;
- at each output step of a plurality of output steps for a second-pass decoder:
- determining, using a same attention mechanism, both:
- an acoustic context vector that summarizes the encoded acoustic frames;
- and
- a linguistic context vector based on a sequence of decoded labels previously output by the second-pass decoder;
- determining, using the acoustic context vector, as output from the second-pass decoder, an acoustic-based probability distribution over possible output labels;
- determining, using the linguistic context vector, as output from the second-pass decoder, a text-based probability distribution over possible output labels;
- and
- interpolating the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels, wherein the possible output labels in the acoustic-based probability distribution and the possible output labels in the text-based probability distribution comprise a same set subword units;
- and
- determining a transcription of the utterance based on the interpolating of the acoustic-based probability distribution over possible output labels and the text-based probability distribution over possible output labels at each of the plurality of output steps.
Dependencies are inferred from the claim text.