Method and apparatus for multilingual and multi-speaker speech synthesis

An apparatus comprising:

• a memory configured to store instructions;
• at least one processor configured, by executing the instructions, to:
• receive training text and a training audio signal obtained from a predetermined speaker uttering the training text;
• identify a language identifier corresponding to the training text and a training reference audio signal obtained from the speaker uttering reference text different from the training text;
• and
• train a speech synthesis model by using training samples that include the training text, the training audio signal, the language identifier, and the training reference audio signal,
• wherein the speech synthesis model comprises:
• a language embedding module configured to transform the language identifier into a language embedding;
• and
• a speaker encoder configured to transform the training audio signal and the training reference audio signal into speaker embeddings, and
• wherein the at least one processor is further configured to:
• determine a loss based on a first speaker embedding transformed from the training reference audio signal, a second speaker embedding transformed from the training audio signal, and the language embedding;
• and
• update parameters of the speaker encoder using the loss.

A method comprising:

• receiving a speech synthesis request including input text and information on a target speaker;
• identifying a language identifier corresponding to the input text and a reference audio signal obtained from the target speaker uttering reference text different from the input text;
• and
• generating, by applying the input text, the language identifier, and the reference audio signal to a speech synthesis model, a synthesized audio signal that simulates speech by the target speaker uttering the input text;
• wherein generating the synthesized audio signal includes:
• transforming, by a language embedding module of the speech synthesis model, the language identifier into a target language embedding;
• and
• transforming, by a speaker encoder of the speech synthesis model, the reference audio signal into a target speaker embedding, and
• wherein the speaker encoder has been trained by using a loss determined based on speaker embeddings transformed from a plurality of training audio signals from a speaker and a language embedding transformed from training text corresponding to one of the plurality of training audio signals.