One stage text-to-speech system with prosody control

A computing system for training a machine-learned expressive speech audio generation system for use in a video game, wherein the system is configured to:

• receive one or more training examples, each training example comprising:
• (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;
• for each of the one or more training examples:
• receive prosody data corresponding to the reference speech signal;
• process, by a speech encoder, the reference speech signal to generate a speech feature embedding;
• process, by a text encoder, the reference text data to generate a text feature embedding;
• generate, by an embedding converter, one or more embedding conversions, wherein, in generating, the system is configured to:
• process, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding;
• and/or
• process, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;
• the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;
• process, by a prosody encoder, the prosody data to generate a prosody feature embedding;
• generate, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;
• determine a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions;
• and
• update one or more of the following:
• the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.

One or more non-transitory computer-readable storage media comprising instructions which, when executed by one or more processors, cause the one or more processors to carry out a method comprising:

• receiving one or more training examples, each training example comprising:
• (i) a reference speech signal and (ii) reference text data corresponding to the reference speech signal;
• for each of the one or more training examples:
• receiving prosody data corresponding to the reference speech signal;
• processing, by a speech encoder, the reference speech signal to generate a speech feature embedding;
• processing, by a text encoder, the reference text data to generate a text feature embedding;
• generating, by an embedding converter, one or more embedding conversions, wherein generating comprises:
• processing, by the embedding converter, the speech feature embedding to generate a speech converted text feature embedding;
• and/or
• processing, by the embedding converter, the text feature embedding to generate a text converted speech feature embedding;
• the one or more embedding conversions comprising the speech converted text feature embedding and/or the text converted speech feature embedding;
• processing, by a prosody encoder, the prosody data to generate a prosody feature embedding;
• generating, by a speech decoder, a reconstructed speech signal based upon the speech feature embedding and the prosody feature embedding;
• determining a loss value using a loss function based upon the reconstructed speech signal, the reference speech signal and the one or more embedding conversions, wherein the loss function comprises a conversion loss term based upon an error measure of the generated one or more embedding conversions;
• and
• updating one or more of the following:
• the speech decoder, the prosody encoder, the speech encoder, the embedding converter, the speech encoder, or the text encoder based upon the determined loss value.