System and method for cross-speaker style transfer in text-to-speech and training data generation

A method implemented by a computing system for training a voice conversion machine learning model within a voice conversion module to generate a spectrogram for a target speaker with a new prosody style of a source speaker, the method comprising:

• training a phonetic posterior gram (PPG) to spectrogram component of the voice conversion machine learning model, wherein the PPG to spectrogram component of the voice conversion machine learning model, during training, is initially trained on multi-speaker data and is configured for converting PPG data into spectrogram data;
• refining the PPG to spectrogram component of the voice conversion machine learning model with target speaker data from a target speaker having a particular voice timber and particular prosody style, by adapting the PPG to spectrogram component to convert PPG data to spectrogram data having the particular prosody style of the target speaker;
• receiving electronic content comprising new PPG data converted from a waveform of source speaker data, wherein the new PPG data is aligned to the waveform of the source speaker data;
• receiving one or more prosody features extracted from the waveform of the source speaker data;
• and
• applying the source speaker data to the voice conversion machine learning model, wherein the refined PPG to spectrogram component of the voice conversion machine learning model is configured to generate a spectrogram in the particular voice timbre of the target speaker, but having the new prosody style of the source speaker rather than the particular prosody style of the target speaker.

A method implemented by a computing system for generating training data for training a neural text-to-speech (TTS) model configured to generate speech data from arbitrary text, the method comprising:

• receiving electronic content comprising source speaker data from a source speaker;
• converting a waveform of the source speaker data to phonetic posterior gram (PPG) data;
• extracting one or more prosody features from the source speaker data;
• applying at least the PPG data and the one or more extracted prosody features to a pre-trained PPG to spectrogram component of a voice conversion module, the pre-trained PPG to spectrogram component configured to generate a spectrogram in a particular voice timbre of a target speaker, but having a new prosody style of the source speaker rather than a particular prosody style of the target speaker;
• and
• generating training data configured to train a neural TTS model, the training data comprising a plurality of spectrograms characterized by the particular voice timbre of the target speaker and the new prosody style of the source speaker.