# US 10,373,605 B2 · Artificial intelligence-based text-to-speech system and method

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,373,605 B2
- Document ID: US10373605B2
- Status: Granted
- Kind: B2
- Application number: 16/022,823
- Filed: 2018-06-29
- Published: 2018-11-22
- Granted: 2019-08-06
- Earliest claimed: 2017-05-18
- Assignees: Telepathy Labs, Inc.
- Applicants: Telepathy Labs, Inc.
- Inventors: Martin Reber; Vijeta Avijeet
- Web page: https://patentgrep.com/patent/US10373605B2

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G06F 18/10: Pre-processing; Data cleansing
- G06F 18/2135: based on approximation criteria, e.g. principal component analysis
- G06F 18/217: {Validation; Performance evaluation; Active pattern learning techniques}
- G06N 3/02: Neural networks
- G06N 3/042: Knowledge-based neural networks; Logical representations of neural networks
- G06N 3/0495: Quantised networks; Sparse networks; Compressed networks
- G06N 3/0499: Feedforward networks
- G06N 3/08: Learning methods
- G06N 3/09: Supervised learning
- G06N 5/02: Knowledge representation; Symbolic representation
- G10L 13/04: Details of speech synthesis systems, e.g. synthesiser structure or memory management
- G10L 19/00: Speech or audio signals analysis-synthesis techniques for redundancy reduction, e.g. in vocoders; Coding or decoding of speech or audio signals, using source filter models or psychoacoustic analysis (in musical instruments G10H)
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
- G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N)
- G06N: COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS

## Abstract

A technique improves training and speech quality of a text-to-speech (TTS) system having an artificial intelligence, such as a neural network. The TTS system is organized as a front-end subsystem and a back-end subsystem. The front-end subsystem is configured to provide analysis and conversion of text into input vectors, each having at least a base frequency, f0, a phenome duration, and a phoneme sequence that is processed by a signal generation unit of the back-end subsystem. The signal generation unit includes the neural network interacting with a pre-existing knowledgebase of phenomes to generate audible speech from the input vectors. The technique applies an error signal from the neural network to correct imperfections of the pre-existing knowledgebase of phenomes to generate audible speech signals. A back-end training system is configured to train the signal generation unit by applying psychoacoustic principles to improve quality of the generated audible speech signals.

## Claims

### Claim 1

A text-to-speech (TTS) training system comprising:
a subsystem configured to receive an input vector from conversion of text, the subsystem including a neural network interacting with a pre-existing knowledgebase of phonemes to apply an error signal to correct for speech signal distortions of the pre-existing knowledgebase of phonemes to generate a corrected speech signal; and
a training subsystem coupled to the subsystem, the training subsystem configured to iteratively correct the subsystem for the speech signal distortions of the pre-existing knowledgebase of phonemes based on psychoacoustic processing for the subsystem to apply the error signal to correct for the speech signal distortions of the pre-existing knowledgebase of phonemes to generate the corrected speech signal, wherein the training subsystem is further configured to ignore inaudible errors of the corrected speech signal based on masking.

### Claim 2 (depends on 1)

The TTS training system of claim 1 wherein the training subsystem is further configured to calculate audible errors of the corrected speech signal by the psychoacoustic processing.

### Claim 3 (depends on 1)

The TTS training system of claim 1 wherein masking includes one of frequency masking and temporal masking.

### Claim 4 (depends on 1)

The TTS training system of claim 1 wherein the training subsystem is further configured to iteratively modify the neural network to correct the subsystem.

### Claim 5 (depends on 1)

The TTS training system of claim 1 further comprising a psychoacoustic generator configured to analyze a reference audio signal to determine masking information.

### Claim 6 (depends on 5)

The TTS training system of claim 5 wherein the psychoacoustic generator is further configured to identify locations and energy levels that are audible and inaudible.

### Claim 7 (depends on 1)

The TTS training system of claim 1 further comprising a quality indicator calculator configured to determine a quality indicator based on audible errors using the psychoacoustic processing to ignore inaudible errors.

### Claim 8 (depends on 7)

The TTS training system of claim 7 wherein the quality indicator is calculated based on a total of audible error signal energy.

### Claim 9 (depends on 7)

The TTS training system of claim 7 wherein the iterative correction further comprises a modification of the neural network that is performed so that a total audible error signal energy is below a quality threshold.

### Claim 10 (depends on 9)

The TTS training system of claim 9 wherein the quality indicator at least converges close to zero with the modification of the neural network.

### Claim 11

A method of training text-to-speech (TTS) processing comprising:
receiving, by a subsystem, an input vector from conversion of text;
interacting, by a neural network of the subsystem, with a pre-existing knowledgebase of phonemes to apply an error signal to correct for speech signal distortions of the pre-existing knowledgebase of phonemes to generate a corrected speech signal;
iteratively correcting, by a training subsystem coupled to the subsystem, the subsystem for the speech signal distortions of the pre-existing knowledgebase of phonemes based on psychoacoustic processing for the subsystem to apply the error signal to correct for the speech signal distortions of the pre-existing knowledgebase of phonemes to generate the corrected speech signal; and
ignoring, by the training subsystem, inaudible errors of the corrected speech signal based on masking.

### Claim 12 (depends on 11)

The method of training TTS processing of claim 11 further comprising calculating audible errors of the corrected speech signal by the psychoacoustic processing.

### Claim 13 (depends on 11)

The method of training TTS processing of claim 11 wherein masking includes one of frequency masking and temporal masking.

### Claim 14 (depends on 11)

The method of training TTS processing of claim 11 wherein the iterative correction further comprises modifying the neural network to correct the subsystem.

### Claim 15 (depends on 11)

The method of training TTS processing of claim 11 further comprising analyzing a reference audio signal to determine masking information.

### Claim 16 (depends on 11)

The method of training TTS processing of claim 11 further comprising identifying locations and energy levels that are audible and inaudible.

### Claim 17 (depends on 11)

The method of training TTS processing of claim 11 further comprising: determining a quality indicator based on audible errors; and ignoring inaudible errors using the psychoacoustic processing.

### Claim 18 (depends on 17)

The method of training TTS processing of claim 17 wherein the quality indicator is calculated based on a total of audible error signal energy.

### Claim 19 (depends on 17)

The method of training TTS processing of claim 17 wherein the iterative correction further comprises modifying the neural network so that a total audible error signal energy is below a quality threshold.

### Claim 20

A non-transitory computer-readable medium having program instructions for training text-to-speech (TTS) processing which, when executed across one or more processors, causes at least a portion of the one or more processors to perform operations comprising:
receiving, by a subsystem, an input vector from conversion of text;
interacting, by a neural network of the subsystem, with a pre-existing knowledgebase of phonemes to apply an error signal to correct for speech signal distortions of the pre-existing knowledgebase of phonemes to generate a corrected speech signal;
iteratively correcting, by a training subsystem coupled to the subsystem, the subsystem for the speech signal distortions of the pre-existing knowledgebase of phonemes based on psychoacoustic processing to apply the error signal to correct for the speech signal distortions of the pre-existing knowledgebase of phonemes to generate the corrected speech signal; and
ignoring, by the training subsystem, inaudible errors of the corrected speech signal based on masking.