Knowledge distillation with domain mismatch for speech recognition

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving distillation data comprising a plurality of out-of-domain training utterances;
• for each particular out-of-domain training utterance of the distillation data:
• generating a corresponding augmented out-of-domain training utterance;
• generating, using a teacher automated speech recognition (ASR) model trained on training data corresponding to a target domain, a corresponding probability distribution over possible speech recognition hypotheses for the corresponding augmented out-of-domain training utterance;
• and
• generating multiple corresponding pseudo labels that represent the N-best speech recognition hypotheses from the corresponding probability distribution over possible speech recognition hypotheses;
• and
• distilling a student ASR model from the teacher ASR model by, for each corresponding augmented out-of-domain training utterance:
• generating, using the student ASR model, a transcription corresponding to the corresponding augmented out-of-domain training utterance;
• generating a loss computed over the multiple corresponding pseudo-labels weighted by the corresponding probability distribution over possible speech recognition results and the transcription corresponding to the corresponding augmented out-of-domain training utterance;
• and
• updating parameters of the teacher model using the loss.