Building a Text-to-Speech System from a Small Amount of Speech Data

A system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving a first plurality of recorded speech samples from an assortment of speakers and a second plurality of recorded speech samples from a target speaker, the assortment of speakers not including the target speaker;
• training a text-to-speech (TTS) model using the first plurality of recorded speech samples from the assortment of speakers, the second plurality of recorded speech samples from the target speaker corresponds to less than fifty percent of the retraining speech data, the trained TTS model configured to output synthetic speech as an audible representation of a text input;
• and
• re-training the trained TTS model using retraining speech data, the retraining speech data comprising the second plurality of recorded speech samples from the target speaker combined with the first plurality of recorded speech samples from the assortment of speakers, the re-trained TTS model configured to output synthetic speech resembling speaking characteristics of the target speaker.