Systems and methods for using neural codec language model for zero-shot text-to-speech synthesis

A method for generating modified personalized speech, the method comprising:

• accessing a machine learning model configured as a zero-shot cross-lingual text-to-speech model which has been previously trained on a text-to-speech training dataset;
• obtaining a text prompt;
• obtaining a speech sample comprising audio data from a target speaker, wherein the target speaker is an unseen target speaker such that no audio data from the target speaker was included in the text-to-speech training dataset;
• accessing an attribute ID configured to modify personalized speech output according to a particular attribute;
• applying the text prompt, the speech sample, and the attribute ID to the machine learning model;
• and
• generating personalized speech output based on the text prompt in a synthesized voice of the target speaker using the speech sample, the synthesized voice of the target speaker being modified according to the attribute ID.

A method for generating personalized speech output, the method comprising:

• accessing a machine learning model configured as a zero-shot cross-lingual text-to-speech model which has been previously trained on a speech-to-speech training dataset comprising a plurality of different bilingual speech transcription pairs;
• obtaining a speech sample comprising audio data from a target speaker, wherein the target speaker is an unseen target speaker such that no audio data from the target speaker was included in the text-to-speech training dataset;
• generating a first transcription of the speech sample in a first language;
• generating a second transcription of the speech sample in a second language based on translating the first transcription of the speech sample into the second language;
• applying the first transcription of the speech sample, the second transcription of the speech sample, and the speech sample to the machine learning model;
• and
• generating a personalized speech output based on the second transcription of the speech sample in the second language using a synthesized voice of the target speaker based on the speech sample from the target speaker.