Contrastive Siamese network for semi-supervised speech recognition

A contrastive Siamese network for training a speech recognition model, the contrastive Siamese network comprising an unsupervised subnetwork trained on a plurality of unlabeled audio samples corresponding to spoken utterances not paired with corresponding transcriptions, the unsupervised subnetwork comprising:

• a target branch configured to:
• receive, as input to an audio encoder of the speech recognition model, a sequence of acoustic frames extracted from the unlabeled audio samples;
• and
• at each of a plurality of time steps, generate a target branch output for a corresponding acoustic frame in the sequence of acoustic frames input to the audio encoder at the corresponding time step;
• and
• an augmented branch configured to:
• perform augmentation on the sequence of acoustic frames extracted from the unlabeled audio samples to generate a sequence of augmented acoustic frames;
• at each of the plurality of time steps, generate, as output from the audio encoder, a higher order feature representation for a corresponding augmented acoustic frame in the sequence of augmented acoustic frames;
• and
• at each of the plurality of time steps, generate, using the higher order feature representation output from the audio encoder at the corresponding time step, a prediction of the target branch output generated by the target branch at the corresponding time step,
• wherein the unsupervised subnetwork is configured to:
• at each of the plurality of time steps, determine an unsupervised loss term based on the target branch output generated by the target branch at the corresponding time step and the prediction of the target branch output generated by the augmented branch at the corresponding time step;
• and
• update parameters of the audio encoder based on the unsupervised loss term determined at each of the plurality of time steps.