Consistency Prediction On Streaming Sequence Models

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a set of training utterance pairs, each training utterance pair comprising:
• a non-synthetic speech representation of a corresponding utterance;
• and
• a synthetic speech representation of the corresponding utterance;
• at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs:
• generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance;
• generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance;
• and
• determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses;
• and
• updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs.