Training A Language Model Of An End-To-End Automatic Speech Recognition Model Using Random Encoder Features

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
• obtaining a training text sample, the training text sample not paired with corresponding audio data;
• generating a sequence of pseudo-random encoder variables;
• processing, using a decoder of a sequence transduction model, the sequence of pseudo-random encoder variables to predict a probability distribution over possible output labels;
• determining a loss metric based on the training text sample and the predicted probability distribution over possible output labels;
• and
• training the decoder based on the loss metric.