Cross-lingual voice conversion system and method

A method of training a cycle-consistent generative adversarial network (CycleGAN) system comprising:

• simultaneously learning a forward mapping function and an inverse mapping function using at least adversarial loss and cycle-consistency loss functions, the forward mapping function comprising:
• receiving, by a voice feature extractor, a first voice audio segment in a first language;
• extracting, by the voice feature extractor, first-voice, speaker-dependent acoustic features;
• sending the first-voice, speaker-dependent acoustic features to a first-to-third speaker generator of the CycleGAN system;
• receiving, by the first-to-third speaker generator, second-voice, speaker-independent linguistic features from the inverse mapping function;
• generating, by the first-to-third speaker generator, a third voice candidate using the first-voice, speaker-dependent acoustic features and the second-voice, speaker-independent linguistic features;
• and
• determining, by a first discriminator of the CycleGAN system, whether there is a discrepancy between the third voice candidate and the first-voice, speaker-dependent acoustic features;
• and the inverse mapping function comprising:
• receiving, by the feature extractor, a second voice audio segment in a second language;
• extracting, by the feature extractor, the second-voice, speaker-independent linguistic features;
• sending the second-voice, speaker-independent linguistic features to a second-to-third voice candidate generator;
• receiving, by the second-to-third voice candidate generator, first-voice, speaker-dependent acoustic features from the forward mapping function;
• generating, by the second-to-third voice candidate generator, a third voice candidate using the second-voice, speaker-independent linguistic features and first-voice, speaker-dependent acoustic features;
• and
• determining, by a second discriminator, whether there is a discrepancy between the third voice candidate and the second-voice, speaker-independent linguistic features.

A machine learning system stored in memory of a server computer system and being implemented by at least one processor, the machine learning system comprising:

• a voice feature extractor configured to:
• receive a first voice audio segment in a first language and a second voice audio segment in a second language, and
• extract, respectively from the first voice and second voice audio segments, audio features comprising first-voice, speaker-dependent acoustic features and second-voice, speaker-independent linguistic features;
• and
• a generative adversarial network (GAN) comprising one or more generators and one or more discriminators, the one or more generators configured to:
• receive extracted features, and
• produce therefrom a third voice candidate having the first-voice, speaker-dependent acoustic features and the second-voice, speaker-independent linguistic features, wherein the third voice candidate speaks the second language;
• and
• the one or more discriminators configured to:
• compare the third voice candidate with ground truth data comprising the first-voice, speaker-dependent acoustic features and second-voice, speaker-independent linguistic features;
• and
• provide results of the comparing back to the generator for refining the third voice candidate.