End-to-end text-to-speech conversion

One or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating, from a sequence of characters in a particular natural language, a spectrogram of a verbal utterance of the sequence of characters in the particular natural language using a text-to-speech conversion system, the operations comprising:

• processing, using an encoder neural network of the text-to-speech conversion system, the sequence of characters to generate a respective encoded representation of each of the characters in the sequence, comprising, for each character in the sequence of characters:
• receiving a respective embedding of the character in the sequence,
• processing the respective embedding of the character in the sequence to generate a respective transformed embedding of the character, and
• processing the respective transformed embedding of the character in the sequence to generate a respective encoded representation of the character;
• receiving a sequence of decoder inputs;
• for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the spectrogram;
• and
• generating a waveform from the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

A system comprising:

• one or more computers;
• and
• one or more non-transitory computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for generating, from a sequence of characters in a particular natural language, a spectrogram of a verbal utterance of the sequence of characters in the particular natural language using a text-to-speech conversion system, the operations comprising:
• processing, using an encoder neural network of the text-to-speech conversion system, the sequence of characters to generate a respective encoded representation of each of the characters in the sequence, comprising, for each character in the sequence of characters:
• receiving a respective embedding of the character in the sequence,
• processing the respective embedding of the character in the sequence to generate a respective transformed embedding of the character, and
• processing the respective transformed embedding of the character in the sequence to generate a respective encoded representation of the character;
• receiving a sequence of decoder inputs;
• for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the spectrogram;
• and
• generating a waveform from the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.