System and method for cross-speaker style transfer in text-to-speech and training data generation
AssigneeMicrosoft Technology Licensing, LLC
InventorsShifeng Pan, Lei He, Yulin Li, Sheng Zhao, Chunling Ma
Life of this patent
Estimating the term…
Abstract
Systems are configured for generating spectrogram data characterized by a voice timbre of a target speaker and a prosody style of source speaker by converting a waveform of source speaker data to phonetic posterior gram (PPG) data, extracting additional prosody features from the source speaker data, and generating a spectrogram based on the PPG data and the extracted prosody features. The systems are configured to utilize/train a machine learning model for generating spectrogram data and for training a neural text-to-speech model with the generated spectrogram data.
Claims20
A method implemented by a computing system for generating a spectrogram for a target speaker in a prosody style of a source speaker and for training a machine learning model based on the spectrogram, the method comprising:
- receiving electronic content comprising source speaker data from the source speaker;
- converting a waveform of the source speaker data to phonetic posterior gram (PPG) data, wherein said converting includes aligning the waveform to the PPG data;
- generating a spectrogram based on the PPG data, wherein the spectrogram is characterized by a determined prosody style of the source speaker;
- and
- generating speech data from arbitrary text using a neural text-to-speech (TTS) model that is trained with the generated spectrogram, the speech data being characterized by the prosody style of the source speaker.
A method implemented by a computing system for training a voice conversion machine learning model within a voice conversion module to generate a spectrogram for a target speaker with a new prosody style of a source speaker, the method comprising:
- training a phonetic posterior gram-(PPG)-to-spectrogram component of the voice conversion machine learning model to convert PPG data into spectrogram data;
- refining the PPG-to-spectrogram component with target speaker data from a target speaker having a particular prosody style, said refining being performed by adapting the PPG-to-spectrogram component to convert PPG data to spectrogram data having the particular prosody style;
- receiving electronic content comprising new PPG data converted from a waveform of source speaker data, the new PPG data being aligned to the waveform of the source speaker data;
- and
- applying the source speaker data to the voice conversion machine learning model, wherein the refined PPG-to-spectrogram component is configured to generate a spectrogram having the new prosody style of the source speaker rather than the particular prosody style of the target speaker.
A method implemented by a computing system for generating training data for training a neural text-to-speech (TTS) model configured to generate speech data from arbitrary text, the method comprising:
- receiving electronic content comprising source speaker data from a source speaker;
- converting a waveform of the source speaker data to phonetic posterior gram (PPG) data, wherein said converting includes aligning the waveform to the PPG data;
- applying the PPG data to a pre-trained PPG-to-spectrogram component of a voice conversion module, the pre-trained PPG-to-spectrogram component configured to generate a spectrogram having a new prosody style of the source speaker rather than a particular prosody style of a target speaker;
- generating training data configured to train a neural TTS model, the training data comprising a plurality of spectrograms characterized by the new prosody style of the source speaker;
- and
- generating speech data from arbitrary text using the neural TTS model that is trained on the generated training data, said generating of the speech data is performed using cross-speaker style transfer, and the speech data is characterized by the new prosody style of the source speaker.
Dependencies are inferred from the claim text.