Systems and methods for neural voice cloning with a few samples

A generative text-to-speech system comprising:

• one or more processors;
• and
• a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
• given a limited set of one or more audios of a new speaker that was not part of training data used to train a neural multi-speaker generative model, using a speaker encoder model comprising a first set of trained model parameters to obtain a speaker embedding for the new speaker given the limited set of one or more audios as an input to the speaker encoder model;
• and
• using the neural multi-speaker generative model comprising a second set of trained model parameters, an input text, and the speaker embedding for the new speaker generated by the speaker encoder model comprising the first set of trained model parameters to generate a synthesized audio representation for the input text in which the synthesized audio includes speech characteristics of the new speaker,
• wherein the neural multi-speaker generative model comprising the second set of trained parameters was trained using as inputs, for a speaker, (1) a training set of text-audio pairs, in which a text-audio pair comprises a text and a corresponding audio of that text by the speaker, and (2) a speaker embedding corresponding to a speaker identifier for that speaker.

A computer-implemented method for synthesizing audio from an input text, comprising:

• receiving a limited set of one or more texts and corresponding ground truth audios of a new speaker that was not part of training data used to train a neural multi- speaker generative model, which training results in speaker embedding parameters for a set of speaker embeddings;
• inputting the limited set of one or more texts and corresponding ground truth audios for the new speaker and at least one or more of the speaker embeddings comprising speaker embedding parameters into the neural multi-speaker generative model comprising pre-trained model parameters or trained model parameters;
• using a comparison of a synthesized audio generated by the neural multi-speaker generative model to its corresponding ground truth audio to adjust at least some of the speaker embedding parameters to obtain a speaker embedding that represents speaker characteristics of the new speaker;
• and
• using the neural multi-speaker generative model comprising trained model parameters, the input text, and the speaker embedding for the new speaker to generate a synthesized audio representation for the input text in which the synthesized audio includes speaker characteristics of the new speaker.

A generative text-to-speech system comprising:

• one or more processors;
• and
• a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
• receiving a limited set of one or more texts and corresponding ground truth audios of a new speaker that was not part of training data used to train a neural multi-speaker generative model, which training results in speaker embedding parameters for a set of speaker embeddings;
• inputting the limited set of one or more texts and corresponding ground truth audios for the new speaker and at least one or more of the speaker embeddings comprising speaker embedding parameters into the neural multi-speaker generative model comprising pre-trained model parameters or trained model parameters;
• using a comparison of a synthesized audio generated by the neural multi-speaker generative model to its corresponding ground truth audio to adjust at least some of the speaker embedding parameters to obtain a speaker embedding that represents speaker characteristics of the new speaker;
• and
• using the neural multi-speaker generative model comprising trained model parameters, the input text, and the speaker embedding for the new speaker to generate a synthesized audio representation for the input text in which the synthesized audio includes speaker characteristics of the new speaker.