Systems and methods for text-to-speech synthesis

A system for text to speech synthesis, the system comprising:

• a memory that stores a plurality of processor executable instructions;
• a data interface that receives an input text, a reference spectrogram, and an emotion identifier (emotion ID);
• and
• one or more hardware processors that read and execute the plurality of processor-executable instructions from the memory to perform operations comprising:
• generating, via a first encoder, a vector representation of the input text, generating, via a second encoder, a vector representation of the reference spectrogram, generating, via a variance adaptor, a modified vector representation based on a combined representation including a sum of the vector representation of the input text, the vector representation of the reference spectrogram, and an embedding of the emotion ID, wherein the variance adaptor adapts the combined representation to adjust a pitch, an energy, and a duration and generating, via a decoder, an audio waveform based on the modified vector representation.

A non-transitory machine-readable medium comprising a plurality of machine executable instructions which, when executed by one or more processors, are adapted to cause the one or more processors to perform operations comprising:

• receiving, via a data interface, an input text, a reference spectrogram, and an emotion identifier (emotion ID);
• generating, via a first encoder, a vector representation of the input text;
• generating, via a second encoder, a vector representation of the reference spectrogram;
• generating, via a variance adaptor, a modified vector representation based on a combined representation including a sum of the vector representation of the input text, the vector representation of the reference spectrogram, and an embedding of the emotion ID, wherein the variance adaptor adapts the combined representation to adjust a pitch, an energy, and a duration;
• and
• generating, via a decoder, an audio waveform based on the modified vector representation.