# US 2023/0274728 A1 · Controlling Expressivity In End-to-End Speech Synthesis Systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2023/0274728 A1
- Document ID: US20230274728A1
- Status: Published application
- Kind: A1
- Application number: 18/314,556
- Filed: 2023-05-09
- Published: 2023-08-31
- Earliest claimed: 2019-08-03
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Daisy Stanton; Eric Dean Battenberg; Russell John Wyatt Skerry-Ryan; Soroosh Mariooryad; David Teh-hwa Kao; Thomas Edward Bagby; Sean Matthew Shannon
- Web page: https://patentgrep.com/patent/US20230274728A1

## CPC

- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 25/30: using neural networks
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 13/047: Architecture of speech synthesisers
- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A system for generating an output audio signal includes a context encoder, a text-prediction network, and a text-to-speech (TTS) model. The context encoder is configured to receive one or more context features associated with current input text and process the one or more context features to generate a context embedding associated with the current input text. The text-prediction network is configured to process the current input text and the context embedding to predict, as output, a style embedding for the current input text. The style embedding specifies a specific prosody and/or style for synthesizing the current input text into expressive speech. The TTS model is configured to process the current input text and the style embedding to generate an output audio signal of expressive speech of the current input text. The output audio signal has the specific prosody and/or style specified by the style embedding.

## Claims

### Claim 1

A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations comprising:
receiving, from a query-response system, current input text to be synthesized into expressive speech by a text-to-speech (TTS) model, the current input text corresponding to a response to a current query in a sequence of queries received at the query-response system;
obtaining one or more context features associated with the current input text, the one or more context features comprising audio features associated with one or more queries preceding the current query in the sequence of queries received at the query-response system;
predicting, using a text-prediction network, a style embedding for the current input text based on the one or more context features associated with the current input text, the style embedding specifying a specific style for synthesizing the current input text into express speech; and
generating, using the TTS model, an output audio signal of expressive speech of the current input text by processing the style embedding and the current input text, the output audio signal having the specific style specified by the style embedding.

### Claim 2 (depends on 1)

The method of claim 1, wherein the operations further comprise generating, using a context model, a context embedding associated with the current input text by processing the one or more context features associated with the current input text.

### Claim 3 (depends on 1)

The method of claim 1, wherein the one or more context features associated with the current input text further comprise the response to the current query.

### Claim 4 (depends on 1)

The method of claim 1, wherein the one or more context features associated with the current input text further comprise at least one of:
previous speech synthesized from previous text that precedes the current query; or
upcoming text from the query-response system that follows the current input text.

### Claim 5 (depends on 1)

The method of claim 1, wherein the one or more context features associated with the current input text further comprises a previous style embedding predicted by a text-prediction network based on previous text that precedes the current query and a previous context embedding associated with the previous text.

### Claim 6 (depends on 1)

The method of claim 1, wherein the TTS model comprises:
an encoder neural network configured to:
receive the current input text from the text source; and
process the current input text to generate a respective encoded sequence of the current input text;
a concatenator configured to:
receive the respective encoded sequence of the current input text from the encoder neural network;
receive the style embedding predicted by the textual-prediction network; and
generate a concatenation between the respective encoded sequence of the current input text and the style embedding; and
an attention-based decoder recurrent neural network configured to:
receive a sequence of decoder inputs; and
for each decoder input in the sequence, process the corresponding decoder input and the concatenation between the respective encoded sequence of the current input text and the style embedding to generate r frames of the output audio signal, wherein r comprises an integer greater than one.

### Claim 7 (depends on 6)

The method of claim 6, wherein the encoder neural network comprises:
an encoder pre-net neural network configured to:
receive a respective embedding of each character in a sequence of characters of the current input text; and
for each character, process the respective embedding to generate a respective transformed embedding of the character; and
an encoder CBHG neural network configured to:
receive the transformed embeddings generated by the encoder pre-net neural network; and
process the transformed embeddings to generate the respective encoded sequence of the current input text.

### Claim 8 (depends on 7)

The method of claim 7, wherein the encoder CBHG neural network comprises a bank of 1-D convolutional filters, followed by a highway network, and followed by a bidirectional recurrent neural network.

### Claim 9 (depends on 1)

The method of claim 1, wherein the text-prediction network comprises:
a time-aggregating gated recurrent unit (GRU) recurrent neural network (RNN) configured to:
receive the context embedding associated with the current input text and an encoded sequence of the current input text; and
generate a fixed-length feature vector by processing the context embedding and the encoded sequence; and
one or more fully-connected layers configured to predict the style embedding by processing the fixed-length feature vector.

### Claim 10 (depends on 9)

The system of claim 9, wherein the one or more fully-connected layers comprise one or more hidden fully-connected layers using ReLU activations and an output layer that uses tanh activation to emit the predicted style embedding.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations comprising:
receiving, from a query-response system, current input text to be synthesized into expressive speech by a text-to-speech (TTS) model, the current input text corresponding to a response to a current query in a sequence of queries received at the query-response system;
obtaining one or more context features associated with the current input text, the one or more context features comprising audio features associated with one or more queries preceding the current query in the sequence of queries received at the query-response system;
predicting, using a text-prediction network, a style embedding for the current input text based on the one or more context features associated with the current input text, the style embedding specifying a specific style for synthesizing the current input text into express speech; and
generating, using the TTS model, an output audio signal of expressive speech of the current input text by processing the style embedding and the current input text, the output audio signal having the specific style specified by the style embedding.

### Claim 12 (depends on 11)

The system of claim 11, wherein the operations further comprise generating, using a context model, a context embedding associated with the current input text by processing the one or more context features associated with the current input text.

### Claim 13 (depends on 11)

The system of claim 11, wherein the one or more context features associated with the current input text further comprise the response to the current query.

### Claim 14 (depends on 11)

The system of claim 11, wherein the one or more context features associated with the current input text further comprise at least one of:
previous speech synthesized from previous text that precedes the current query; or
upcoming text from the query-response system that follows the current input text.

### Claim 15 (depends on 11)

The system of claim 11, wherein the one or more context features associated with the current input text further comprises a previous style embedding predicted by a text-prediction network based on previous text that precedes the current query and a previous context embedding associated with the previous text.

### Claim 16 (depends on 11)

The system of claim 11, wherein the TTS model comprises:
an encoder neural network configured to:
receive the current input text from the text source; and
process the current input text to generate a respective encoded sequence of the current input text;
a concatenator configured to:
receive the respective encoded sequence of the current input text from the encoder neural network;
receive the style embedding predicted by the textual-prediction network; and
generate a concatenation between the respective encoded sequence of the current input text and the style embedding; and
an attention-based decoder recurrent neural network configured to:
receive a sequence of decoder inputs; and
for each decoder input in the sequence, process the corresponding decoder input and the concatenation between the respective encoded sequence of the current input text and the style embedding to generate r frames of the output audio signal, wherein r comprises an integer greater than one.

### Claim 17 (depends on 16)

The system of claim 16, wherein the encoder neural network comprises:
an encoder pre-net neural network configured to:
receive a respective embedding of each character in a sequence of characters of the current input text; and
for each character, process the respective embedding to generate a respective transformed embedding of the character; and
an encoder CBHG neural network configured to:
receive the transformed embeddings generated by the encoder pre-net neural network; and
process the transformed embeddings to generate the respective encoded sequence of the current input text.

### Claim 18 (depends on 17)

The system of claim 17, wherein the encoder CBHG neural network comprises a bank of 1-D convolutional filters, followed by a highway network, and followed by a bidirectional recurrent neural network.

### Claim 19 (depends on 11)

The system of claim 11, wherein the text-prediction network comprises:
a time-aggregating gated recurrent unit (GRU) recurrent neural network (RNN) configured to:
receive the context embedding associated with the current input text and an encoded sequence of the current input text; and
generate a fixed-length feature vector by processing the context embedding and the encoded sequence; and
one or more fully-connected layers configured to predict the style embedding by processing the fixed-length feature vector.

### Claim 20 (depends on 19)

The system of claim 19, wherein the one or more fully-connected layers comprise one or more hidden fully-connected layers using ReLU activations and an output layer that uses tanh activation to emit the predicted style embedding.
