# US 2024/0112667 A1 · Synthesis of speech from text in a voice of a target speaker using neural networks

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2024/0112667 A1
- Document ID: US20240112667A1
- Status: Published application
- Kind: A1
- Application number: 18/525,475
- Filed: 2023-11-30
- Published: 2024-04-04
- Earliest claimed: 2018-05-17
- Assignees: Google Inc.
- Applicants: Google LLC
- Inventors: Ye Jia; Zhifeng Chen; Yonghui Wu; Jonathan Shen; Ruoming Pang; Ron J. Weiss; Ignacio Lopez Moreno; Fei Ren; Yu Zhang; Quan Wang; Patrick An Phu Nguyen
- Web page: https://patentgrep.com/patent/US20240112667A1

## CPC

- G10L 13/033: Voice editing, e.g. manipulating the voice of the synthesiser
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G06N 3/0442: characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/09: Supervised learning
- G06N 3/096: Transfer learning
- G10L 17/04: Training, enrolment or model building
- G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H)
- G10L 25/30: using neural networks
- G06N 3/08: Learning methods
- G10L 2013/021: {Overlap-add techniques}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for speech synthesis. The methods, systems, and apparatus include actions of obtaining an audio representation of speech of a target speaker, obtaining input text for which speech is to be synthesized in a voice of the target speaker, generating a speaker vector by providing the audio representation to a speaker encoder engine that is trained to distinguish speakers from one another, generating an audio representation of the input text spoken in the voice of the target speaker by providing the input text and the speaker vector to a spectrogram generation engine that is trained using voices of reference speakers to generate audio representations, and providing the audio representation of the input text spoken in the voice of the target speaker for output.

## Claims

### Claim 1

A computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations comprising:
obtaining an audio recording file comprising speech of a target speaker;
extracting, using a speaker encoder network, from the audio recording file, a speaker embedding vector for the target speaker;
obtaining input text to be synthesized in a voice of the target speaker;
generating, using a synthesizer configured to receive the speaker embedding vector and the input text as input, a synthesized speech representation of the input text spoken in the voice of the target speaker; and
providing the synthesized speech representation of the input text spoken in the voice of the target speaker for output.

### Claim 2 (depends on 1)

The method of claim 1, wherein the speaker encoder network is trained to extract speaker embedding vectors from audio representations corresponding to utterances spoken by the same speaker that are close together in an embedding space.

### Claim 3 (depends on 1)

The method of claim 1, wherein the speaker encoder network is trained to extract speaker embedding vectors from audio representations corresponding to utterances spoken by different speakers that are distant from each other.

### Claim 4 (depends on 1)

The method of claim 1, wherein the speaker encoder network comprises a long short-term memory (LSTM) neural network.

### Claim 5 (depends on 1)

The method of claim 1, wherein the speaker encoder network is trained separately from the synthesizer.

### Claim 6 (depends on 5)

The method of claim 5, wherein, during training of the synthesizer, parameters of the speaker encoder network are fixed.

### Claim 7 (depends on 1)

The method of claim 1, wherein the synthesizer comprises a spectrogram generation neural network that is trained to predict mel spectrograms from a sequence of phoneme inputs.

### Claim 8 (depends on 7)

The method of claim 7, wherein the spectrogram generation neural network comprises a sequence-to-sequence attention neural network.

### Claim 9 (depends on 7)

The method of claim 7, wherein the spectrogram generation neural network comprises an encoder neural network and a decoder neural network.

### Claim 10 (depends on 9)

The method of claim 9, wherein the spectrogram generation neural network further comprises an attention layer.

### Claim 11

A system comprising:
data processing hardware; and
memory hardware in communication with the data processing hardware and storing instructions that when executed on the data processing hardware cause the data processing hardware to perform operations comprising:
obtaining an audio recording file comprising speech of a target speaker;
extracting, using a speaker encoder network, from the audio recording file, a speaker embedding vector for the target speaker;
obtaining input text to be synthesized in a voice of the target speaker;
generating, using a synthesizer configured to receive the speaker embedding vector and the input text as input, a synthesized speech representation of the input text spoken in the voice of the target speaker; and
providing the synthesized speech representation of the input text spoken in the voice of the target speaker for output.

### Claim 12 (depends on 11)

The system of claim 11, wherein the speaker encoder network is trained to extract speaker embedding vectors from audio representations corresponding to utterances spoken by the same speaker that are close together in an embedding space.

### Claim 13 (depends on 11)

The system of claim 11, wherein the speaker encoder network is trained to extract speaker embedding vectors from audio representations corresponding to utterances spoken by different speakers that are distant from each other.

### Claim 14 (depends on 11)

The system of claim 11, wherein the speaker encoder network comprises a long short-term memory (LSTM) neural network.

### Claim 15 (depends on 11)

The system of claim 11, wherein the speaker encoder network is trained separately from the synthesizer.

### Claim 16 (depends on 15)

The system of claim 15, wherein, during training of the synthesizer, parameters of the speaker encoder network are fixed.

### Claim 17 (depends on 11)

The system of claim 11, wherein the synthesizer comprises a spectrogram generation neural network that is trained to predict mel spectrograms from a sequence of phoneme inputs.

### Claim 18 (depends on 17)

The system of claim 17, wherein the spectrogram generation neural network comprises a sequence-to-sequence attention neural network.

### Claim 19 (depends on 17)

The system of claim 17, wherein the spectrogram generation neural network comprises an encoder neural network and a decoder neural network.

### Claim 20 (depends on 19)

The system of claim 19, wherein the spectrogram generation neural network further comprises an attention layer.
