# US 12,190,860 B2 · End-to-end text-to-speech conversion

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 12,190,860 B2
- Document ID: US12190860B2
- Status: Granted
- Kind: B2
- Application number: 18/516,069
- Filed: 2023-11-21
- Published: 2024-04-18
- Granted: 2025-01-07
- Earliest claimed: 2017-03-29
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Samuel Bengio; Yuxuan Wang; Zongheng Yang; Zhifeng Chen; Yonghui Wu; Ioannis Agiomyrgiannakis; Ron J. Weiss; Navdeep Jaitly; Ryan M. Rifkin; Robert Andrew James Clark; Quoc V. Le; Russell J. Ryan; Ying Xiao
- Web page: https://patentgrep.com/patent/US12190860B2

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 15/16: using artificial neural networks
- G10L 25/18: the extracted parameters being spectral information of each sub-band
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating speech from text. One of the systems includes one or more computers and one or more storage devices storing instructions that when executed by one or more computers cause the one or more computers to implement: a sequence-to-sequence recurrent neural network configured to: receive a sequence of characters in a particular natural language, and process the sequence of characters to generate a spectrogram of a verbal utterance of the sequence of characters in the particular natural language; and a subsystem configured to: receive the sequence of characters in the particular natural language, and provide the sequence of characters as input to the sequence-to-sequence recurrent neural network to obtain as output the spectrogram of the verbal utterance of the sequence of characters in the particular natural language.

## Claims

### Claim 1

A computer-implemented method for generating, from a representation of a text input, a mel-scale spectrogram of a verbal utterance of the text input using a text-to-speech conversion system, the method comprising:
processing, using an encoder neural network of the text-to-speech conversion system, the representation of the text input to generate a respective encoded representation of each of a plurality of components in the representation of the text input;
receiving a sequence of decoder inputs; and
for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the mel-scale spectrogram,
wherein the encoder neural network and the decoder neural network have been trained on training data that maps representations of text inputs to mel-scale spectrograms.

### Claim 2 (depends on 1)

The method of claim 1, wherein the encoder neural network comprises an encoder pre-net neural network and an encoder CBHG neural network, and
wherein processing, using the encoder neural network of the text-to-speech conversion system, the representation of the text input to generate a respective encoded representation of each of the plurality of components in the representation of the text input comprises:
receiving, using the encoder pre-net neural network, a respective embedding of each component in the plurality of the components,
processing, using the encoder pre-net neural network, the respective embedding of each component in the plurality of components to generate a respective transformed embedding of the component, and
processing, using the encoder CBHG neural network, a respective transformed embedding of each component in the plurality of components to generate a respective encoded representation of the component.

### Claim 3 (depends on 2)

The method of claim 2, wherein the encoder CBHG neural network comprises a bank of 1-D convolutional filters, followed by a highway network, and followed by a bidirectional recurrent neural network.

### Claim 4 (depends on 3)

The method of claim 3, wherein the bidirectional recurrent neural network is a gated recurrent unit neural network.

### Claim 5 (depends on 3)

The method of claim 3, wherein the encoder CBHG neural network includes a residual connection between the transformed embeddings and outputs of the bank of 1-D convolutional filters.

### Claim 6 (depends on 3)

The method of claim 3, wherein the bank of 1-D convolutional filters includes a max pooling along time layer with stride one.

### Claim 7 (depends on 1)

The method of claim 1, wherein a first decoder input in the sequence of decoder inputs is a predetermined initial frame.

### Claim 8 (depends on 1)

The method of claim 1, further comprising:
processing the mel-scale spectrogram to generate a waveform synthesizer input; and
processing, using a waveform synthesizer of the text-to-speech conversion system, the waveform synthesizer input to generate a waveform of the verbal utterance of the text input.

### Claim 9 (depends on 1)

The method of claim 1, further comprising generating a waveform from the mel-scale spectrogram of the verbal utterance of the text input.

### Claim 10 (depends on 9)

The method of claim 9, further comprising:
generating speech using the waveform; and
providing the generated speech for playback.

### Claim 11 (depends on 8)

The method of claim 8, wherein the waveform synthesizer is a trainable spectrogram to waveform inverter.

### Claim 12 (depends on 8)

The method of claim 8, wherein the waveform synthesizer is a vocoder.

### Claim 13 (depends on 8)

The method of claim 8, wherein the waveform synthesizer input is a linear-scale spectrogram of the verbal utterance of the text input.

### Claim 14

A system comprising:
one or more computers; and
one or more non-transitory computer storage media storing instructions that, when executed by the one or more computers, cause the one or more computers to perform operations for generating, from a representation of a text input, a mel-scale spectrogram of a verbal utterance of the text input using a text-to-speech conversion system, the operations comprising:
processing, using an encoder neural network of the text-to-speech conversion system, the representation of the text input to generate a respective encoded representation of each of a plurality of components in the representation of the text input;
receiving a sequence of decoder inputs; and
for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the mel-scale spectrogram,
wherein the encoder neural network and the decoder neural network have been trained on training data that maps representations of text inputs to mel-scale spectrograms.

### Claim 15 (depends on 14)

The system of claim 14, wherein the encoder neural network comprises an encoder pre-net neural network and an encoder CBHG neural network, and
wherein processing, using the encoder neural network of the text-to-speech conversion system, the representation of the text input to generate a respective encoded representation of each of the plurality of components comprises:
receiving, using the encoder pre-net neural network, a respective embedding of each component in the plurality of components,
processing, using the encoder pre-net neural network, the respective embedding of each component in the plurality of components to generate a respective transformed embedding of the component, and
processing, using the encoder CBHG neural network, a respective transformed embedding of component in the plurality of components to generate a respective encoded representation of the component.

### Claim 16 (depends on 14)

The system of claim 14, wherein the operations further comprise generating a waveform from the mel-scale spectrogram of the verbal utterance of the text input in a particular natural language.

### Claim 17 (depends on 16)

The system of claim 16, wherein the operations further comprises:
generating speech using the waveform; and
providing the generated speech for playback.

### Claim 18

One or more non-transitory computer-readable storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations for generating, from a representation of a text input, a mel-scale spectrogram of a verbal utterance of the text input using a text-to-speech conversion system, the operations comprising:
processing, using an encoder neural network of the text-to-speech conversion system, the representation of the text input to generate a respective encoded representation of each of a plurality of components in the representation of the text input;
receiving a sequence of decoder inputs; and
for each decoder input in the sequence of decoder inputs, processing, using a decoder neural network of the text-to-speech conversion system, the decoder input and the encoded representations to generate multiple frames of the mel-scale spectrogram,
wherein the encoder neural network and the decoder neural network have been trained on training data that maps representations of text inputs to mel-scale spectrograms.

### Claim 19 (depends on 18)

The one or more non-transitory computer-readable storage media of claim 18, wherein the operations further comprise generating a waveform from the mel-scale spectrogram of the verbal utterance of the text input in a particular natural language.

### Claim 20 (depends on 19)

The one or more non-transitory computer-readable storage media of claim 19, wherein the operations further comprises:
generating speech using the waveform; and
providing the generated speech for playback.