# US 11,908,448 B2 · Parallel tacotron non-autoregressive and controllable TTS

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 11,908,448 B2
- Document ID: US11908448B2
- Status: Granted
- Kind: B2
- Application number: 17/327,076
- Filed: 2021-05-21
- Published: 2022-04-21
- Granted: 2024-02-20
- Earliest claimed: 2020-10-21
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Isaac Elias; Jonathan Shen; Yu Zhang; Ye Jia; Ron J. Weiss; Yonghui Wu; Byungha Chun
- Web page: https://patentgrep.com/patent/US11908448B2

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G06F 40/126: Character encoding
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/088: Non-supervised learning, e.g. competitive learning
- G06N 3/09: Supervised learning
- G10L 13/047: Architecture of speech synthesisers
- G06N 3/048: Activation functions
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A method for training a non-autoregressive TTS model includes receiving training data that includes a reference audio signal and a corresponding input text sequence. The method also includes encoding the reference audio signal into a variational embedding that disentangles the style/prosody information from the reference audio signal and encoding the input text sequence into an encoded text sequence. The method also includes predicting a phoneme duration for each phoneme in the input text sequence and determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration. The method also includes generating one or more predicted mel-frequency spectrogram sequences for the input text sequence and determining a final spectrogram loss based on the predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence. The method also includes training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss.

## Claims

### Claim 1

A computer-implemented method when executed on data processing hardware causes the data processing hardware to perform operations for training a non-autoregressive text-to-speech (TTS) model, the operations comprising:
receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;
encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal;
encoding, using a text encoder, the input text sequence into an encoded text sequence;
predicting, using a duration decoder comprising a stack of self-attention blocks followed by two independent projections, based on the encoded text sequence and the variational embedding, a phoneme duration for each phoneme in the input text sequence by:
predicting, using a sigmoid activation following a first one of the two independent projections, a probability of non-zero duration for each phoneme;
predicting, using a softplus activation following a second one of the two independent projections, the phoneme duration for each phoneme;
determining whether the probability of non-zero duration predicted for the corresponding phoneme is less than a threshold value; and
when the probability of non-zero duration is less than the threshold value, zeroing out the phoneme duration predicted for the corresponding phoneme;
determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence
generating, as output from a non-autoregressive spectrogram decoder comprising a stack of self-attention blocks, based on an output of the duration decoder, multiple predicted mel-frequency spectrogram sequences for the input text sequence;
determining a final spectrogram loss based on the multiple predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence sampled from the reference audio signal; and
training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss determined for each phoneme in the input text sequence.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1, wherein each self-attention block in the stack of self-attention blocks comprises an identical lightweight convolution (LConv) block.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1, wherein each self-attention block in the stack of self-attention blocks comprises an identical transformer block.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1, wherein the input text sequence comprises words each having one or more phonemes, silences at all word boundaries, and punctuation marks.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1, wherein:
the residual encoder comprises a global variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and
encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1, wherein:
the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal;
aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and
encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1, wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:
a gated linear unit (GLU) layer;
a LConv layer configured to receive an output of the GLU layer;
a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and
a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

### Claim 8 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
upsampling, using the reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence, the output of the duration decoder into a number of frames; and
obtaining positional embeddings representing phoneme position information for each phoneme in the input text utterance,
wherein generating the multiple predicted mel-frequency spectrogram sequences for the input text sequence is based on the positional embeddings and the upsampling of the output of the duration decoder into the number of frames.

### Claim 9 (depends on 1)

The computer-implemented method of claim 1, wherein:
generating the multiple predicted mel-frequency spectrogram sequences for the input text sequence comprises generating, as output from each self-attention block in the stack of self-attention blocks of the spectrogram decoder, a respective mel-frequency spectrogram sequence; and
determining the final spectrogram loss comprises:
for each respective predicted mel-frequency spectrogram sequence, determining a respective spectrogram loss based on the predicted mel-frequency spectrogram sequence and the reference mel-frequency spectrogram sequence; and
aggregating the respective spectrogram losses determined for the predicted mel-frequency spectrogram sequences to generate the final spectrogram loss.

### Claim 10 (depends on 1)

The computer-implemented method of claim 1, wherein the operations further comprise:
concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and
generating the output of the duration decoder based on the duration decoder receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

### Claim 11 (depends on 1)

The computer-implemented method of claim 1, wherein:
the input text sequence includes a sequence of phonemes; and
encoding the input text sequence into the encoded text sequence comprises:
receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;
for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;
processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and
processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.

### Claim 12

A system for training a non-autoregressive text-to-speech (TTS) model, the system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware, the memory hardware storing instructions that when executed by the data processing hardware cause the data processing hardware to perform operations comprising:
receiving training data including a reference audio signal and a corresponding input text sequence, the reference audio signal comprising a spoken utterance and the input text sequence corresponds to a transcript of the reference audio signal;
encoding, using a residual encoder, the reference audio signal into a variational embedding, the variational embedding disentangling style/prosody information from the reference audio signal;
encoding, using a text encoder, the input text sequence into an encoded text sequence;
predicting, using a duration decoder comprising a stack of self-attention blocks followed by two independent projections, based on the encoded text sequence and the variational embedding, a phoneme duration for each phoneme in the input text sequence by:
predicting, using a sigmoid activation following a first one of the two independent projections, a probability of non-zero duration for each phoneme;
predicting, using a softplus activation following a second one of the two independent projections, the phoneme duration for each phoneme;
determining whether the probability of non-zero duration predicted for the corresponding phoneme is less than a threshold value; and
when the probability of non-zero duration is less than the threshold value, zeroing out the phoneme duration predicted for the corresponding phoneme;
determining a phoneme duration loss based on the predicted phoneme durations and a reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence
generating, as output from a non-autoregressive spectrogram decoder comprising a stack of self-attention blocks, based on an output of the duration decoder, multiple predicted mel-frequency spectrogram sequences for the input text sequence;
determining a final spectrogram loss based on the multiple predicted mel-frequency spectrogram sequences and a reference mel-frequency spectrogram sequence sampled from the reference audio signal; and
training the TTS model based on the final spectrogram loss and the corresponding phoneme duration loss determined for each phoneme in the input text sequence.

### Claim 13 (depends on 12)

The system of claim 12, wherein each self-attention block in the stack of self-attention blocks comprises an identical lightweight convolution (LConv) block.

### Claim 14 (depends on 12)

The system of claim 12, wherein each self-attention block in the stack of self-attention blocks comprises an identical transformer block.

### Claim 15 (depends on 12)

The system of claim 12, wherein the input text sequence comprises words each having one or more phonemes, silences at all word boundaries, and punctuation marks.

### Claim 16 (depends on 12)

The system of claim 12, wherein:
the residual encoder comprises a global variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal; and
encoding, using the global VAE, the reference mel-frequency spectrogram sequence into the variational embedding.

### Claim 17 (depends on 12)

The system of claim 12, wherein:
the residual encoder comprises a phoneme-level fine-grained variational autoencoder (VAE); and
encoding the reference audio signal into the variational embedding comprises:
sampling the reference mel-frequency spectrogram sequence from the reference audio signal;
aligning the reference mel-frequency spectrogram sequence with each phoneme in a sequence of phonemes extracted from the input text sequence; and
encoding, using the phoneme-level fine-grained VAE, based on aligning the reference mel-frequency spectrogram sequence with each phoneme in the sequence of phonemes, a sequence of phoneme-level variational embeddings.

### Claim 18 (depends on 12)

The system of claim 12, wherein the residual encoder comprises a stack of lightweight convolution (LConv) blocks, each LConv block in the stack of LConv blocks comprises:
a gated linear unit (GLU) layer;
a LConv layer configured to receive an output of the GLU layer;
a residual connection configured to concatenate an output of the LConv layer with an input to the GLU layer; and
a final feedforward layer configured to receive, as input, the residual connection concatenating the output of the LConv layer with the input to the GLU layer.

### Claim 19 (depends on 12)

The system of claim 12, wherein the operations further comprise:
upsampling, using the reference phoneme duration sampled from the reference audio signal for each phoneme in the input text sequence, the output of the duration decoder into a number of frames; and
obtaining positional embeddings representing phoneme position information for each phoneme in the input text utterance,
wherein generating the multiple predicted mel-frequency spectrogram sequences for the input text sequence is based on the positional embeddings and the upsampling of the output of the duration decoder into the number of frames.

### Claim 20 (depends on 12)

The system of claim 12, wherein:
generating the multiple predicted mel-frequency spectrogram sequences for the input text sequence comprises generating, as output from each self-attention block in the stack of self-attention blocks of the spectrogram decoder, a respective mel-frequency spectrogram sequence; and
determining the final spectrogram loss comprises:
for each respective predicted mel-frequency spectrogram sequence, determining a respective spectrogram loss based on the predicted mel-frequency spectrogram sequence and the reference mel-frequency spectrogram sequence; and
aggregating the respective spectrogram losses determined for the predicted mel-frequency spectrogram sequences to generate the final spectrogram loss.

### Claim 21 (depends on 12)

The system of claim 12, wherein the operations further comprise:
concatenating the encoded text sequence, the variational embedding, and a reference speaker embedding representing an identity of a reference speaker that uttered the reference audio signal; and
generating the output of the duration decoder based on the duration decoder receiving, as input, the concatenation of the encoded text sequence, the variational embedding, and the reference speaker embedding.

### Claim 22 (depends on 12)

The system of claim 12, wherein:
the input text sequence includes a sequence of phonemes; and
encoding the input text sequence into the encoded text sequence comprises:
receiving, from a phoneme look-up table, a respective embedding of each phoneme in the sequence of phonemes;
for each phoneme in the sequence of phonemes, processing, using an encoder pre-net neural network of the text encoder, the respective embedding to generate a respective transformed embedding of the phoneme;
processing, using a bank of convolutional blocks, the respective transformed embeddings to generate convolution outputs; and
processing, using a stack of self-attention blocks, the convolution outputs to generate the encoded text sequence.