Monte Carlo Self-Training for Speech Recognition

A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations for training a sequence transduction model, the operations comprising:

• receiving, as input to a self-training network comprising an unsupervised subnetwork trained on a plurality of unlabeled input samples, a sequence of unlabeled input features extracted from the unlabeled input samples;
• using a teacher branch comprising a teacher encoder of the unsupervised subnetwork:
• processing the sequence of unlabeled input features to predict probability distributions over possible teacher branch output labels;
• sampling, from the predicted probability distributions over possible teacher branch output labels, one or more sequences of teacher branch output labels;
• and
• determining a sequence of pseudo output labels based on the one or more sequences of teacher branch output labels sampled from the predicted probability distributions over possible teacher branch output labels;
• and
• using a student branch comprising a student encoder of the unsupervised subnetwork:
• processing the sequence of unlabeled input features extracted from the unlabeled input samples to predict probability distributions over possible student branch output labels;
• determining a negative log likelihood term based on the predicted probability distributions over possible student branch output labels and the sequence of pseudo output labels;
• and
• updating parameters of the student encoder based on the negative log likelihood term.