Chunk-wise attention for longform ASR

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving training data comprising:
• a corpus of multilingual unspoken textual utterances, each unspoken textual utterance not paired with any corresponding spoken utterance of non-synthetic speech;
• a corpus of multilingual un-transcribed non-synthetic speech utterances, each un-transcribed non-synthetic speech utterance not paired with a corresponding transcription;
• and
• a corpus of multilingual transcribed non-synthetic speech utterances, each transcribed non-synthetic speech utterance paired with a corresponding transcription;
• for each corresponding un-transcribed non-synthetic speech utterance in the corpus of multilingual un-transcribed non-synthetic speech utterances:
• generating, at each of a plurality of output steps, using a random-projection quantizer, a target quantized vector token and a target token index for a corresponding audio feature in a sequence of audio features associated with the corresponding un-transcribed non-synthetic speech utterance, wherein the target token index maps the corresponding audio feature to the target quantized vector token stored in one or more codebooks;
• after masking a subset of the audio features in the sequence of audio features associated with the corresponding un-transcribed non-synthetic speech utterance, generating, by an audio encoder, contrastive context vectors from corresponding masked audio features;
• and
• deriving a contrastive loss term between the contrastive context vectors at the masked positions and the target token index;
• generating, using an alignment model, a corresponding alignment output for each unspoken textual utterance;
• at each of a plurality of output steps for each alignment output:
• generating, using an auxiliary decoder, a first probability distribution over possible speech recognition hypotheses for the corresponding alignment output;
• and
• determining an alignment output loss term based on the first probability distribution over possible speech recognition hypotheses and the unspoken textual utterance corresponding to the alignment output;
• at each of a plurality of output steps for each transcribed non-synthetic speech utterance:
• generating, using the auxiliary decoder, a second probability distribution over possible speech recognition hypotheses for the corresponding transcribed non-synthetic speech utterance;
• and
• determining a non-synthetic speech loss term based on the second probability distribution over possible speech recognition hypotheses and the corresponding transcription paired with the transcribed non-synthetic speech utterance;
• and
• pre-training the audio encoder based on the contrastive loss term, the alignment output loss term, and the non-synthetic speech loss term.