Unified speech representation learning

A method comprising:

• obtaining a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set, as well as latent speech representations based on the first training data set;
• training a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set;
• applying the latent speech representations from the machine learning model to a transformer context network to generate contextual representations;
• aligning each contextual representation included in the contextual representations with a phoneme label to generate phonetically-aware contextual representations;
• causing a refinement engine to further refine the machine learning model based on a refinement dataset, wherein the refinement engine fine-tunes the machine learning model on a limited labeled dataset corresponding to a low-resource target language and/or target domain;
• and
• transforming at least some of the contextual representations by randomly replacing a sub-set of the contextual representations with quantized latent speech representations.

One or more hardware storage devices that store instructions that are executable by one or more processors to cause the one or more processors to:

• obtain a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set, as well as latent speech representations based on the first training data set;
• train a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set;
• apply the latent speech representations from the machine learning model to a transformer context network to generate contextual representations;
• align each contextual representation included in the contextual representations with a phoneme label to generate phonetically-aware contextual representations;
• cause a refinement engine to further refine the machine learning model based on a refinement dataset, wherein the refinement engine fine-tunes the machine learning model on a limited labeled dataset corresponding to a low-resource target language and/or target domain;
• and
• transform at least some of the contextual representations by randomly replacing a sub-set of the contextual representations with quantized latent speech representations.