# US 10,872,596 B2 · Systems and methods for parallel wave generation in end-to-end text-to-speech

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,596 B2
- Document ID: US10872596B2
- Status: Granted
- Kind: B2
- Application number: 16/277,919
- Filed: 2019-02-15
- Published: 2019-06-13
- Granted: 2020-12-22
- Earliest claimed: 2017-10-19
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Wei Ping; Kainan Peng; Jitong Chen
- Web page: https://patentgrep.com/patent/US10872596B2

## CPC

- G10L 13/00: Speech synthesis; Text to speech systems
- G06F 9/30003: {Arrangements for executing specific machine instructions}
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Described herein are embodiments of an end-to-end text-to-speech (TTS) system with parallel wave generation. In one or more embodiments, a Gaussian inverse autoregressive flow is distilled from an autoregressive WaveNet by minimizing a novel regularized Kullback-Leibler (KL) divergence between their highly-peaked output distributions. Embodiments of the methodology computes the KL divergence in a closed-form, which simplifies the training process and provides very efficient distillation. Embodiments of a novel text-to-wave neural architecture for speech synthesis are also described, which are fully convolutional and enable fast end-to-end training from scratch. These embodiments significantly outperform the previous pipeline that connects a text-to-spectrogram model to a separately trained WaveNet. Also, a parallel waveform synthesizer embodiment conditioned on the hidden representation in an embodiment of this end-to-end model were successfully distilled.

## Claims

### Claim 1

A text-to-speech system comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
converting, using an encoder, textual features of an input text into encoder hidden representations;
decoding, using a decoder, the encoder hidden representations with attention into frame-level hidden representations in an autoregressive manner;
processing, using a convolutional processing block, the frame-level hidden representations into sample-level hidden representations; and
synthesizing, using a non-autoregressive distilled vocoder distilled from an autoregressive vocoder, waveforms corresponding to the input text conditioned on the sample-level hidden representations.

### Claim 2 (depends on 1)

The text-to-speech system of claim 1 wherein the non-autoregressive distilled vocoder is an inverse autoregressive flow (IAF) trained using at least a regularized Kullback-Leibler (KL) divergence between output distributions of the autoregressive vocoder and the non-autoregressive distilled vocoder, the regularized KL divergence incorporates a regularization in addition to a reverse KL divergence to stabilize the training process.

### Claim 3 (depends on 2)

The text-to-speech system of claim 2 wherein the output distribution of the autoregressive vocoder is a single Gaussian distribution.

### Claim 4 (depends on 2)

The text-to-speech system of claim 2 wherein the regularized KL divergence is obtained in a closed-form.

### Claim 5 (depends on 2)

The text-to-speech system of claim 2 wherein the non-autoregressive distilled vocoder is trained using a frame-level loss between output from the non-autoregressive distilled vocoder and corresponding ground-truth audio, in combination with the regularized KL divergence.

### Claim 6 (depends on 5)

The text-to-speech system of claim 5 wherein the frame-level loss is a spectrogram frame loss.

### Claim 7 (depends on 1)

The text-to-speech system of claim 1 wherein the encoder, the decoder, and the convolutional processing block are pre-trained with the autoregressive vocoder to fix the parameters of the encoder, the decoder, and the convolutional processing block.

### Claim 8 (depends on 1)

The text-to-speech system of claim 1 wherein the convolutional processing block is non-causal and enabled to apply non-causal convolution to utilize future temporal information.

### Claim 9

A computer-implemented method for training an end-to-end text-to-speech system to synthesize speech from an input text, comprising:
encoding, via an encoder comprising one or more convolutional blocks, the input text into encoder hidden representations including key representations and value representations;
autoregressively decoding, using a decoder, the encoder hidden representations with attention into frame-level hidden representations;
processing, using a convolutional processing block, the frame-level hidden representations into sample-level hidden representations;
generating, using an autoregressive vocoder, synthesized waveforms corresponding to the input text conditioned on the sample-level hidden representations; and
distilling the autoregressive vocoder to obtain a distilled parallel vocoder based on Gaussian inverse autoregressive flow (IAF) using at least a regularized Kullback-Leibler (KL) divergence between output distributions of the autoregressive vocoder and the distilled parallel vocoder.

### Claim 10 (depends on 9)

The computer-implemented method of claim 9 wherein the encoder, the decoder, and the convolutional processing block are pre-trained and have parameters fixed during the distillation.

### Claim 11 (depends on 9)

The computer-implemented method of claim 9 wherein the output distribution of the autoregressive vocoder is a single Gaussian distribution.

### Claim 12 (depends on 11)

The computer-implemented method of claim 11 wherein the regularized KL divergence is obtained in a closed-form.

### Claim 13 (depends on 9)

The computer-implemented method of claim 9 wherein the distilled parallel vocoder is trained using a frame-level loss between output from the distilled vocoder and corresponding ground-truth audio, in combination with the regularized KL divergence.

### Claim 14 (depends on 9)

The computer-implemented method of claim 9 wherein the distilled parallel vocoder is a non-autoregressive model.

### Claim 15 (depends on 9)

The computer-implemented method of claim 9 wherein the convolutional processing block is non-causal and enabled to apply non-causal convolution to utilize future temporal information.

### Claim 16

A computer-implemented method for training a text-to-speech system to synthesize speech from ground-truth spectrograms, comprising:
receiving, at an autoregressive vocoder, ground-truth spectrograms for waveform synthesizing;
receiving, at a parallel vocoder distilled from the autoregressive vocoder based on Gaussian inverse autoregressive flow (IAF), the ground-truth spectrograms for waveform synthesizing; and
training the parallel vocoder using a loss function having a linear combination of a frame-level loss and a regularized Kullback-Leibler (KL) divergence between waveform distributions of the autoregressive vocoder and the parallel vocoder.

### Claim 17 (depends on 16)

The computer-implemented method of claim 16 wherein the regularized KL divergence incorporates a regularization in addition to a reverse KL divergence to stabilize the training process.

### Claim 18 (depends on 16)

The computer-implemented method of claim 16 wherein the output distribution of the autoregressive vocoder is a single Gaussian distribution.

### Claim 19 (depends on 16)

The computer-implemented method of claim 16 wherein the regularized KL divergence is obtained in a closed-form.

### Claim 20 (depends on 16)

The computer-implemented method of claim 16 wherein the frame-level loss is a spectrogram frame loss obtained, using a ground-truth data set, from difference between output of the parallel vocoder and corresponding ground-truth audio.