Self-training with oracle and top-ranked hypotheses

A system, comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising:
• receiving a plurality of training samples, each training sample of the plurality of training samples comprising:
• a corresponding sequence of acoustic frames characterizing a corresponding training utterance;
• and
• a corresponding ground-truth transcription of the corresponding training utterance;
• for each particular training sample of the plurality of training samples:
• processing, using an automatic speech recognition (ASR) model, the corresponding sequence of acoustic frames to obtain an n-best list of speech recognition hypotheses for the corresponding training utterance;
• for each speech recognition hypothesis of the n-best list of speech recognition hypotheses for the corresponding training utterance, determining a corresponding number of word errors relative to the corresponding ground-truth transcription;
• for a top-ranked hypothesis from the n-best list of speech recognition hypotheses, determining a first loss based on the corresponding ground-truth transcription;
• identifying, as an oracle hypothesis, the speech recognition hypothesis from the n-best list of speech recognition hypotheses having the smallest corresponding number of word errors relative to the corresponding ground-truth transcription;
• determining a second loss for the oracle hypothesis based on the corresponding ground-truth transcription;
• and
• determining a corresponding combined loss based on the first and second losses;
• and
• training the ASR model based on the corresponding combined losses determined for the plurality of training samples.

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:

• receiving a plurality of training samples, each training sample of the plurality of training samples comprising:
• a corresponding sequence of acoustic frames characterizing a corresponding training utterance;
• and
• a corresponding ground-truth transcription of the corresponding training utterance;
• for each particular training sample of the plurality of training samples:
• processing, using an automatic speech recognition (ASR) model, the corresponding sequence of acoustic frames to obtain an n-best list of speech recognition hypotheses for the corresponding training utterance;
• for each speech recognition hypothesis of the n-best list of speech recognition hypotheses for the corresponding training utterance, determining a corresponding number of word errors relative to the corresponding ground-truth transcription;
• for a top-ranked hypothesis from the n-best list of speech recognition hypotheses, determining a first loss based on the corresponding ground-truth transcription;
• scaling the first loss by the corresponding number of word errors determined for the top-ranked hypothesis;
• identifying, as an oracle hypothesis, the speech recognition hypothesis from the n-best list of speech recognition hypotheses having the smallest corresponding number of word errors relative to the corresponding ground-truth transcription;
• determining a second loss for the oracle hypothesis based on the corresponding ground-truth transcription;
• scaling the second loss by the corresponding number of word errors determined for the oracle hypothesis;
• and
• determining a corresponding combined loss based on the scaled first loss and the scaled second loss;
• and
• training the ASR model based on the corresponding combined losses determined for the plurality of training samples.