Coupling speech encoders with downstream text models
AssigneeGoogle Inc.
InventorsCiprian Ioan Chelba, Johan Schalkwyk
Life of this patent
Estimating the term…
Loading patent…
AssigneeGoogle Inc.
InventorsCiprian Ioan Chelba, Johan Schalkwyk
Estimating the term…
A method includes receiving an exporter module training dataset including a plurality of transcribed speech utterances each spoken in a corresponding source language and including acoustic frames paired with a corresponding ground-truth transcription in the corresponding source language. For each transcribed speech utterance, the method also includes processing, using a pre-trained audio encoder, the acoustic frames to generate audio encodings; processing, using a speech decoder, the audio encodings to generate a 1-best sequence of predicted speech recognition labels in the source language; generating, using an exporter module, exporter embeddings by embedding the audio encodings aligned with the 1-best sequence of predicted speech recognition labels; and determining an L2 loss based the exporter embeddings and a sequence of source language embeddings. The method also includes training the exporter module based on the L2 losses determined for the transcribed speech utterances.
A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
A system comprising:
Dependencies are inferred from the claim text.