Speech Recognition Using Unspoken Text and Speech Synthesis

A system for training a speech recognition model, the system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• obtaining a plurality of unspoken text utterances associated with a target domain the speech recognition model is being trained to learn;
• obtaining a set of spoken training utterances, each spoken training utterance comprising a corresponding transcription paired with a corresponding non-synthetic speech representation of the spoken training utterance;
• for each unspoken text utterance:
• conditioning the unspoken text utterance on a randomly assigned speaker embedding from a set of speaker embeddings, each speaker embedding in the set of speaker embeddings representing speaker characteristics;
• and
• generating, as output from a pre-trained text-to-speech (TTS) model configured to receive the corresponding unspoken text utterance as input, a synthetic speech representation of the corresponding unspoken text utterance conditioned on the randomly assigned speaker embedding;
• and
• training the speech recognition model on the synthetic speech representations generated as output from the pre-trained TTS model and the non-synthetic speech representations in the set of spoken training utterances, the speech recognition model comprising frame alignment-based transducer model.