Parallel tacotron non-autoregressive and controllable TTS

A system for training a non-autoregressive text-to-speech (TTS) model, the system comprising:

• data processing hardware;
• and
• memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:
• receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;
• encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal;
• encoding, using a text encoder, the input text sequence into an encoded text sequence;
• predicting, using a duration decoder comprising a stack of self-attention blocks followed by two independent projections, based on the encoded text sequence and the variational embedding, a phoneme duration for each phoneme in the input text sequence by:
• predicting, using a sigmoid activation following a first one of the two independent projections, a probability of non-zero duration for each phoneme;
• predicting, using a softplus activation following a second one of the two independent projections, the phoneme duration for each phoneme;
• determining whether the probability of non-zero duration predicted for the corresponding phoneme is less than a threshold value;
• and
• when the probability of non-zero duration is less than the threshold value, zeroing out the phoneme duration predicted for the corresponding phoneme;
• determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence
• generating, as output from a non-autoregressive spectrogram decoder comprising a stack of self-attention blocks, based on an output of the duration decoder, multiple predicted mel-frequency spectrogram sequences for the input text sequence;
• determining a final spectrogram loss based on the multiple predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence sampled from the reference audio signal;
• and
• training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss determined for each phoneme in the input text sequence.