Using non-parallel voice conversion for speech conversion models

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware and storing instructions that when executed by the data processing hardware causes the data processing hardware to perform operations comprising:
• receiving a set of training utterances each comprising a non-synthetic speech representation of a corresponding utterance;
• for each training utterance, generating a corresponding voice conversion synthetic speech representation by using a voice conversion model to convert the non-synthetic speech representation into the corresponding voice conversion synthetic representation of the corresponding utterance, the voice conversion model comprising a content encoder configured to encode each training utterance, the content encoder comprising a pretrained automatic speech recognition (ASR) encoder, wherein the non-synthetic speech representation and the voice conversion synthetic speech representation form a corresponding training utterance pair;
• at each of a plurality of output steps for each training utterance pair in the set of training utterance pairs:
• generating, for output by a speech recognition model, a first probability distribution over possible non-synthetic speech recognition hypotheses for the corresponding non-synthetic speech representation of the corresponding utterance;
• generating, for output by the speech recognition model, a second probability distribution over possible synthetic speech recognition hypotheses for the corresponding synthetic speech representation of the corresponding utterance;
• and
• determining a consistent loss term for the corresponding training utterance pair based on the first probability distribution over possible non-synthetic speech recognition hypotheses and the second probability distribution over possible non-synthetic speech recognition hypotheses;
• and
• updating parameters of the speech recognition model based on the consistent loss term determined at each of the plurality of output steps for each training utterance pair in the set of training utterance pairs,
• wherein the voice conversion model comprises:
• the content encoder further configured to:
• receive, as input, input speech;
• and
• generate, as output, an initial latent representation;
• a vector quantization-variational autoencoder (VQ-VAE) layer configured to:
• receive, as input, the initial latent representation;
• and
• generate, as output, a latent representation of linguistic content extracted from the input speech that discards speaking style variations in the input speech;
• and
• a decoder conditioned on a speaker embedding and configured to generate output speech based on the latent representation of linguistic content.