# US 10,872,598 B2 · Systems and methods for real-time neural text-to-speech

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,872,598 B2
- Document ID: US10872598B2
- Status: Granted
- Kind: B2
- Application number: 15/882,926
- Filed: 2018-01-29
- Published: 2018-08-30
- Granted: 2020-12-22
- Earliest claimed: 2017-02-24
- Assignees: Baidu USA LLC
- Applicants: Baidu USA, LLC
- Inventors: Sercan O. Arik; Mike Chrzanowski; Adam Coates; Gregory Diamos; Andrew Gibiansky; John Miller; Andrew Ng; Jonathan Raiman; Shubhahrata Sengupta; Mohammad Shoeybi
- Web page: https://patentgrep.com/patent/US10872598B2

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/082: modifying the architecture, e.g. adding, deleting or silencing nodes or connections
- G06N 3/09: Supervised learning
- G10L 13/027: Concept to speech synthesisers; Generation of natural phrases from machine-based concepts (generation of parameters for speech synthesis out of text G10L13/08)
- G10L 25/30: using neural networks
- G06F 40/242: Dictionaries
- G06N 3/02: Neural networks
- G06N 3/047: Probabilistic or stochastic networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)

## Abstract

Embodiments of a production-quality text-to-speech (TTS) system constructed from deep neural networks are described. System embodiments comprise five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For embodiments of the segmentation model, phoneme boundary detection was performed with deep neural networks using Connectionist Temporal Classification (CTC) loss. For embodiments of the audio synthesis model, a variant of WaveNet was created that requires fewer parameters and trains faster than the original. By using a neural network for each component, system embodiments are simpler and more flexible than traditional TTS systems, where each component requires laborious feature engineering and extensive domain expertise. Inference with system embodiments may be performed faster than real time.

## Claims

### Claim 1

A computer-implemented method for training a text-to-speech (TTS) system to synthesize human speech from text, comprising:
training a grapheme-to-phoneme model to convert written text to phonemes corresponding to the written text;
using the trained grapheme-to-phoneme model to convert written text, which is a transcription corresponding to training audio, to phonemes corresponding to the written text and training audio;
using the training audio and the corresponding phonemes to train a segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes;
given a ground truth dataset comprising ground truth written text representing a transcription of ground truth training audio, using the trained grapheme-to-phoneme model to produce phonemes;
given the ground truth training audio and the corresponding phonemes, using the trained segmentation model to produce phoneme durations; and
using the ground truth training audio, the phonemes, the phoneme durations, and fundamental frequencies of the ground truth training audio to train an audio synthesis model that outputs a signal representing synthesized human speech of the ground truth written text.

### Claim 2 (depends on 1)

The computer-implemented method of claim 1 further comprising:
extracting the fundamental frequencies for the ground truth training audio; and
training a phoneme duration and fundamental frequency model using the fundamental frequencies, the phonemes, and the phoneme durations to output for each phoneme:
a phoneme duration;
a probability that the phoneme is voiced; and
a fundamental frequency profile.

### Claim 3 (depends on 1)

The computer-implemented method of claim 1 wherein the phonemes comprise the phoneme previously obtained as an input to train the segmentation model.

### Claim 4 (depends on 1)

The computer-implemented method of claim 1 wherein one or more of the steps of using the trained grapheme-to-phoneme model to convert written text to phonemes and using the trained grapheme-to-phoneme model to produce phonemes comprises:
using a phoneme dictionary or the trained grapheme-to-phoneme model to convert written text to phonemes, where the grapheme-to-phoneme model is trained using training data from a phoneme dictionary to generalize to unseen text.

### Claim 5 (depends on 1)

The computer-implemented method of claim 1 wherein training the segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes comprises using a connectionist temporal classification (CTC) loss to predict sequences of phoneme pairs.

### Claim 6 (depends on 1)

The computer-implemented method of claim 1 wherein the audio synthesis model uses multiple threads and overlapping computation on those threads to produce the synthesized human speech.

### Claim 7 (depends on 1)

The computer-implemented method of claim 1 wherein the fundamental frequency profile for a phoneme is a set of fundamental frequencies values equally spaced in a time domain across the phoneme duration for the phoneme.

### Claim 8

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
training a grapheme-to-phoneme model to convert written text to phonemes corresponding to the written text;
using the trained grapheme-to-phoneme model to convert written text, which is a transcription corresponding to training audio, to phonemes corresponding to the written text and training audio;
using the training audio and the corresponding phonemes to train a segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes;
given a ground truth dataset comprising ground truth written text representing a transcription of ground truth training audio, using the trained grapheme-to-phoneme model to produce phonemes;
given the ground truth training audio and the corresponding phonemes, using the trained segmentation model to produce phoneme durations; and
using the ground truth training audio, the phonemes, the phoneme durations, and fundamental frequencies of the ground truth training audio to train an audio synthesis model that outputs a signal representing synthesized human speech of the ground truth written text.

### Claim 9 (depends on 8)

The non-transitory computer-readable medium or media of claims 8 further comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
extracting the fundamental frequencies for the ground truth training audio; and
training a phoneme duration and fundamental frequency model using the fundamental frequencies, the phonemes, and the phoneme durations to output for each phoneme:
a phoneme duration;
a probability that the phoneme is voiced; and
a fundamental frequency profile.

### Claim 10 (depends on 8)

The non-transitory computer-readable medium or media of claims 8 wherein the phonemes comprise the phoneme previously obtained as an input to train the segmentation model.

### Claim 11 (depends on 8)

The non-transitory computer-readable medium or media of claims 8 wherein one or more of the steps of using the trained grapheme-to-phoneme model to convert written text to phonemes and using the trained grapheme-to-phoneme model to produce phonemes comprise:
using a phoneme dictionary or the trained grapheme-to-phoneme model to convert written text to phonemes, wherein the grapheme-to-phoneme model is trained using training data from a phoneme dictionary to generalize to unseen text.

### Claim 12 (depends on 8)

The non-transitory computer-readable medium or media of claims 8 wherein training the segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes comprises using a connectionist temporal classification (CTC) loss.

### Claim 13 (depends on 8)

The non-transitory computer-readable medium or media of claims 8 wherein the segmentation model is trained to predict sequences of phoneme pairs.

### Claim 14 (depends on 8)

The non-transitory computer-readable medium or media of claims 8 wherein the fundamental frequency profile for a phoneme is a set of fundamental frequencies values equally spaced in a time domain across the phoneme duration for the phoneme.

### Claim 15

A system comprising:
one or more processors; and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
training a grapheme-to-phoneme model to convert written text to phonemes corresponding to the written text;
using the trained grapheme-to-phoneme model to convert written text, which is a transcription corresponding to training audio, to phonemes corresponding to the written text and training audio;
using the training audio and the corresponding phonemes to train a segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes;
given a ground truth dataset comprising ground truth written text representing a transcription of ground truth training audio, using the trained grapheme-to-phoneme model to produce phonemes;
given the ground truth training audio and the corresponding phonemes, using the trained segmentation model to produce phoneme durations; and
using the ground truth training audio, the phonemes, the phoneme durations, and fundamental frequencies of the ground truth training audio to train an audio synthesis model that outputs a signal representing synthesized human speech of the ground truth written text.

### Claim 16 (depends on 15)

The system of claims 15 wherein the non-transitory computer-readable medium or media further comprises one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:
extracting the fundamental frequencies for the ground truth training audio; and
training a phoneme duration and fundamental frequency model using the fundamental frequencies, the phonemes, and the phoneme durations to output for each phoneme:
a phoneme duration;
a probability that the phoneme is voiced; and
a fundamental frequency profile.

### Claim 17 (depends on 16)

The system of claims 16 wherein the segmentation model is trained to predict sequences of phoneme pairs.

### Claim 18 (depends on 15)

The system of claims 15 wherein the phonemes comprise the phoneme previously obtained as an input to train the segmentation model.

### Claim 19 (depends on 15)

The system of claims 15 wherein one or more of the steps of using the trained grapheme-to-phoneme model to convert written text to phonemes and using the trained grapheme-to-phoneme model to produce phonemes comprises:
using a phoneme dictionary or the trained grapheme-to-phoneme model to convert written text to phonemes, wherein the grapheme-to-phoneme model is trained using training data from a phoneme dictionary to generalize to unseen text.

### Claim 20 (depends on 15)

The system of claims 15 wherein training the segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes comprises using a connectionist temporal classification (CTC) loss.