US 10,872,598 B2 · Systems and methods for real-time neural text-to-speech

Abstract

Embodiments of a production-quality text-to-speech (TTS) system constructed from deep neural networks are described. System embodiments comprise five major building blocks: a segmentation model for locating phoneme boundaries, a grapheme-to-phoneme conversion model, a phoneme duration prediction model, a fundamental frequency prediction model, and an audio synthesis model. For embodiments of the segmentation model, phoneme boundary detection was performed with deep neural networks using Connectionist Temporal Classification (CTC) loss. For embodiments of the audio synthesis model, a variant of WaveNet was created that requires fewer parameters and trains faster than the original. By using a neural network for each component, system embodiments are simpler and more flexible than traditional TTS systems, where each component requires laborious feature engineering and extensive domain expertise. Inference with system embodiments may be performed faster than real time.

Claims20

Dependent claims

A computer-implemented method for training a text-to-speech (TTS) system to synthesize human speech from text, comprising:

training a grapheme-to-phoneme model to convert written text to phonemes corresponding to the written text;
using the trained grapheme-to-phoneme model to convert written text, which is a transcription corresponding to training audio, to phonemes corresponding to the written text and training audio;
using the training audio and the corresponding phonemes to train a segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes;
given a ground truth dataset comprising ground truth written text representing a transcription of ground truth training audio, using the trained grapheme-to-phoneme model to produce phonemes;
given the ground truth training audio and the corresponding phonemes, using the trained segmentation model to produce phoneme durations;
and
using the ground truth training audio, the phonemes, the phoneme durations, and fundamental frequencies of the ground truth training audio to train an audio synthesis model that outputs a signal representing synthesized human speech of the ground truth written text.

4 more elements

A non-transitory computer-readable medium or media comprising one or more sequences of instructions which, when executed by at least one processor, causes steps to be performed comprising:

training a grapheme-to-phoneme model to convert written text to phonemes corresponding to the written text;
using the trained grapheme-to-phoneme model to convert written text, which is a transcription corresponding to training audio, to phonemes corresponding to the written text and training audio;
using the training audio and the corresponding phonemes to train a segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes;
given a ground truth dataset comprising ground truth written text representing a transcription of ground truth training audio, using the trained grapheme-to-phoneme model to produce phonemes;
given the ground truth training audio and the corresponding phonemes, using the trained segmentation model to produce phoneme durations;
and
using the ground truth training audio, the phonemes, the phoneme durations, and fundamental frequencies of the ground truth training audio to train an audio synthesis model that outputs a signal representing synthesized human speech of the ground truth written text.

4 more elements

A system comprising:

one or more processors;
and
a non-transitory computer-readable medium or media comprising one or more sets of instructions which, when executed by at least one of the one or more processors, causes steps to be performed comprising:
training a grapheme-to-phoneme model to convert written text to phonemes corresponding to the written text;
using the trained grapheme-to-phoneme model to convert written text, which is a transcription corresponding to training audio, to phonemes corresponding to the written text and training audio;
using the training audio and the corresponding phonemes to train a segmentation model to output phoneme durations by identifying phoneme boundaries in the training audio by aligning it with the corresponding phonemes;
given a ground truth dataset comprising ground truth written text representing a transcription of ground truth training audio, using the trained grapheme-to-phoneme model to produce phonemes;
given the ground truth training audio and the corresponding phonemes, using the trained segmentation model to produce phoneme durations;
and
using the ground truth training audio, the phonemes, the phoneme durations, and fundamental frequencies of the ground truth training audio to train an audio synthesis model that outputs a signal representing synthesized human speech of the ground truth written text.

7 more elements

Dependencies are inferred from the claim text.

Systems and methods for real-time neural text-to-speech

Life of this patent

Systems and methods for real-time neural text-to-speech

Life of this patent

Abstract

Claims20