Exporting modular encoder features for streaming and deliberation ASR

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations comprising:
• obtaining a base encoder from a pre-trained model, the pre-trained model comprising the base encoder and a decoder;
• receiving training data comprising a sequence of acoustic frames characterizing an utterance paired with a ground-truth transcription of the utterance;
• at each corresponding output step of a plurality of output steps:
• generating, by the base encoder, a first encoded representation for a corresponding acoustic frame in the sequence of acoustic frames;
• generating, by an exporter network configured to receive a continuous sequence of first encoded representations generated by the base encoder, a second encoded representation for a corresponding acoustic frame in the sequence of acoustic frames;
• generating, by an exporter decoder, a probability distribution over possible logits;
• and
• determining a corresponding exporter decoder loss based on the ground-truth transcription and the probability distribution over possible logits generated by the exporter decoder at the corresponding output step;
• and
• training the exporter network based on the corresponding exporter decoder loss determined at each corresponding output step of the plurality of output steps while parameters of the base encoder are frozen.