# US 2020/0402497 A1 · Systems and Methods for Speech Generation

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0402497 A1
- Document ID: US20200402497A1
- Status: Published application
- Kind: A1
- Application number: 16/911,314
- Filed: 2020-06-24
- Published: 2020-12-24
- Earliest claimed: 2019-06-24
- Assignees: Replicant Solutions, Inc.
- Applicants: Replicant Solutions, Inc.
- Inventors: Zak Semenov; John Meade; Alessandro Marin; Alexander L. De Souza; Benjamin Gleitzman; Meghna Suresh
- Web page: https://patentgrep.com/patent/US20200402497A1

## CPC

- G10L 13/02: Methods for producing synthetic speech; Speech synthesisers
- G10L 13/10: Prosody rules derived from text; Stress or intonation
- G06N 3/045: Combinations of networks
- G06N 3/0455: Auto-encoder networks; Encoder-decoder networks
- G06N 3/0464: Convolutional networks [CNN, ConvNet]
- G06N 3/0475: Generative networks
- G06N 3/08: Learning methods
- G06N 3/084: Backpropagation, e.g. using gradient descent
- G06N 3/09: Supervised learning
- G06N 3/094: Adversarial learning
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 19/167: {Audio streaming, i.e. formatting and decoding of an encoded audio signal representation into a data stream for transmission or storage purposes}
- G10L 21/0332: involving modification of waveforms
- G06N 3/044: Recurrent networks, e.g. Hopfield networks
- G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H)
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

Systems and methods for generating audio data in accordance with embodiments of the invention are illustrated. One embodiment includes a method for generating audio data. The method includes steps for generating a plurality of style tokens from a set of audio inputs, generating an input feature vector based on the plurality of style tokens and a set of text features, and generating audio data (e.g., a spectrogram, audio waveforms, etc.) based on the input feature vector.

## Claims

### Claim 1

A method for generating audio data, the method comprising:
generating a plurality of style tokens from a set of audio inputs;
generating an input feature vector based on the plurality of style tokens and a set of text features; and
generating audio data based on the input feature vector.

### Claim 2 (depends on 1)

The method of claim 1, wherein generating the plurality of style tokens comprises:
generating a speaker token using a speaker subnetwork; and
generating a prosody token using a prosody subnetwork.

### Claim 3 (depends on 2)

The method of claim 2, wherein at least one of the speaker subnetwork and the prosody subnetwork is a pre-trained network.

### Claim 4 (depends on 1)

The method of claim 1, wherein the set of audio inputs comprises a set of samples with a desired characteristic, wherein the generated audio data reflects the desired characteristic.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the input feature vector comprises at least one of averaging, concatenating, and adding a subset of the plurality of style tokens.

### Claim 6 (depends on 1)

The method of claim 1, wherein the set of text features comprises at least one of raw text, audio data, parts of speech, and phonemes.

### Claim 7 (depends on 1)

The method of claim 1, wherein generating the audio data comprises utilizing a convolution neural network (CNN) to generate a spectrogram.

### Claim 8 (depends on 1)

The method of claim 1, wherein generating the audio data comprises utilizing teacher and student networks to generate the audio data.

### Claim 9 (depends on 8)

The method of claim 8, wherein generating the audio data comprises:
training the teacher network to generate audio data in an autoregressive manner; and
training the student network to learn from the teacher network to generate audio data in a non-autoregressive manner.

### Claim 10 (depends on 9)

The method of claim 9, wherein training the student network comprises training the student network to learn to predict attention from the set of audio inputs, wherein the student network generates the audio data using the predicted attention.

### Claim 11 (depends on 1)

The method of claim 1, wherein the generated audio data is a mel spectrogram.

### Claim 12 (depends on 11)

The method of claim 11, wherein the method further comprises generating audio waveforms from the generated spectrogram.

### Claim 13

A non-transitory machine readable medium containing processor instructions for generating audio data, where execution of the instructions by a processor causes the processor to perform a process that comprises:
generating a plurality of style tokens from a set of audio inputs;
generating an input feature vector based on the plurality of style tokens and a set of text features; and
generating audio data based on the input feature vector.

### Claim 14 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein generating the plurality of style tokens comprises:
generating a speaker token using a speaker subnetwork; and
generating a prosody token using a prosody subnetwork.

### Claim 15 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein the set of audio inputs comprises a set of samples with a desired characteristic, wherein the generated audio data reflects the desired characteristic.

### Claim 16 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein generating the input feature vector comprises at least one of averaging, concatenating, and adding a subset of the plurality of style tokens.

### Claim 17 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein the set of text features comprises at least one of raw text, audio data, parts of speech, and phonemes.

### Claim 18 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein generating the audio data comprises utilizing a convolution neural network (CNN) to generate a spectrogram.

### Claim 19 (depends on 13)

The non-transitory machine readable medium of claim 13, wherein generating the audio data comprises utilizing teacher and student networks to generate the audio data, wherein generating the audio data comprises:
training the teacher network to generate audio data in an autoregressive manner; and
training the student network to learn from the teacher network to generate audio data in a non-autoregressive manner.

### Claim 20 (depends on 9)

The non-transitory machine readable medium of claim 9, wherein training the student network comprises training the student network to learn to predict attention from the set of audio inputs, wherein the student network generates the audio data using the predicted attention.
