# US 2016/0365085 A1 · System and method for outlier identification to remove poor alignments in speech synthesis

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2016/0365085 A1
- Document ID: US20160365085A1
- Status: Published application
- Kind: A1
- Application number: 14/737,080
- Filed: 2015-06-11
- Published: 2016-12-15
- Earliest claimed: 2015-06-11
- Assignees: Interactive Intelligence Group, Inc.
- Applicants: Interactive Intelligence Group, Inc.
- Inventors: E. Veera Raghavendra; Aravind Ganapathiraju
- Web page: https://patentgrep.com/patent/US20160365085A1

## CPC

- G10L 13/08: Text analysis or generation of parameters for speech synthesis out of text, e.g. grapheme to phoneme translation, prosody generation or stress or intonation determination
- G10L 13/00: Speech synthesis; Text to speech systems
- G10L 25/03: characterised by the type of extracted parameters
- G10L 25/51: for comparison or discrimination
- G10L 13/06: Elementary speech units used in speech synthesisers; Concatenation rules
- G10L 13/07: Concatenation rules
- G10L 15/144: {Training of HMMs}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A system and method are presented for outlier identification to remove poor alignments in speech synthesis. The quality of the output of a text-to-speech system directly depends on the accuracy of alignments of a speech utterance. The identification of mis-alignments and mis-pronunciations from automated alignments may be made based on fundamental frequency methods and group delay based outlier methods. The identification of these outliers allows for their removal, which improves the synthesis quality of the text-to-speech system.

## Claims

### Claim 1

A method for identifying outlying results in audio files used for model training, in a text-to-speech system, applying fundamental frequency, the method comprising the steps of:
a. extracting values of the fundamental frequencies from the audio files;
b. generating alignments using the extracted values from the audio files;
c. separating out instances of phonemes;
d. determining, for each separated instance, an average fundamental frequency value and an average duration value;
e. identifying an instance as an outlier, wherein an outlier is identified if:
i. the phoneme is a vowel;
ii. the average fundamental frequency of an instance is less than a predetermined value;
iii. the duration of the instance is greater than twice the average duration of a phoneme; and
iv. the duration of the instance is less than half of the average duration of a phoneme; and
f. identifying a sum of outliers for each sentence in the audio files, wherein if the sentence has more than a number of outliers, discarding the sentence in the audio files from model training.

### Claim 2 (depends on 1)

The method of claim 1, wherein the extracting is performed using a pitch tracking tool.

### Claim 3 (depends on 1)

The method of claim 1, wherein the generating is performed using a speech recognition system.

### Claim 4 (depends on 1)

The method of claim 1, wherein the alignments are generated at a phoneme level.

### Claim 5 (depends on 1)

The method of claim 1, wherein the predetermined value is empirically chosen.

### Claim 6 (depends on 5)

The method of claim 5, wherein the predetermined value is 40.

### Claim 7 (depends on 1)

The method of claim 1, wherein the number of outliers of step (f) is empirically chosen.

### Claim 8 (depends on 7)

The method of claim 7, wherein the number of outliers is five.

### Claim 9

A method for identifying outlying results in audio files used for model training, in a text-to-speech system, applying group delay algorithms, the method comprising the steps of:
a. generating alignments of the audio files at a phoneme level;
b. generating alignments of the audio files at a syllable level;
c. adjusting the alignments at the syllable level using group delay algorithms;
d. separating each syllable from the audio files into a separate audio file;
e. generating, for each separate audio file, phonemes of the separate audio files using phoneme boundaries for each syllable and an existing phoneme model;
f. determining a likelihood value of each generated phoneme, wherein if the likelihood value meets a criteria, identifying the generated phoneme as an outlier; and
g. identifying a sum of outliers for each sentence in the audio files, wherein if the sentence has more than a number of outliers, discarding the sentence from model training.

### Claim 10 (depends on 9)

The method of claim 9, wherein the generating of step (a) is performed using a speech recognition system.

### Claim 11 (depends on 9)

The method of claim 9, wherein the generating of step (b) is performed using at least one of: a speech recognition system and a phoneme model.

### Claim 12 (depends on 9)

The method of claim 9, wherein the criteria comprises a small value.

### Claim 13 (depends on 9)

The method of claim 9, wherein the criteria comprises a failed alignment.

### Claim 14 (depends on 9)

The method of claim 9, wherein the number of outliers of step (g) comprises three.

### Claim 15

A method for synthesizing speech in a text-to-speech system, wherein the system comprises at least a speech database, a database capable of storing Hidden Markov Models, and a synthesis filter, the method comprising the steps of:
a. identifying outlying results in audio files from the speech database and removing the outlying results before model training;
b. converting a speech signal from the speech database into parameters and extracting the parameters from the speech signal;
c. training Hidden Markov Models using the extracted parameters from the speech signal and using the labels from the speech database to produce context dependent Hidden Markov Models;
d. storing the context dependent Hidden Markov Models in the database capable of storing Hidden Markov Models;
e. inputting text and analyzing the text, wherein said analyzing comprises extracting labels from the text;
f. utilizing said labels to generate parameters from the context dependent Hidden Markov Models;
g. generating an other signal from the parameters;
h. inputting the other signal and the parameters into the synthesis filter; and
i. producing synthesized speech as the other signal passes through the synthesis filter.

### Claim 16 (depends on 15)

The method of claim 15, wherein the parameters of step (b) comprise one or more of excitation and spectral.

### Claim 17 (depends on 15)

The method of claim 15, wherein the parameters of step (f) comprise one or more of excitation and spectral.

### Claim 18 (depends on 15)

The method of claim 15, wherein the identifying of outlying results, applying fundamental frequency, in audio files comprises the steps of:
a. extracting values of the fundamental frequencies from the audio files;
b. generating alignments using the extracted values from the audio files;
c. separating out instances of phonemes;
d. determining, for each separated instance, an average fundamental frequency value and an average duration value;
e. identifying an instance as an outlier, wherein an outlier is identified if:
i. the phoneme is a vowel;
ii. the average fundamental frequency of an instance is less than a predetermined value;
iii. the duration of the instance is greater than twice the average duration of a phoneme; and
iv. the duration of the instance is less than half of the average duration of a phoneme;
f. identifying a sum of outliers for each sentence in the audio files, wherein if the sentence has more than a number of outliers, discarding the sentence in the audio files from model training.

### Claim 19 (depends on 18)

The method of claim 18, wherein the extracting is performed using a pitch tracking tool.

### Claim 20 (depends on 18)

The method of claim 18, wherein the generating is performed using a speech recognition system.

### Claim 21 (depends on 18)

The method of claim 18, wherein the alignments are generated at a phoneme level.

### Claim 22 (depends on 18)

The method of claim 18, wherein the predetermined value is empirically chosen.

### Claim 23 (depends on 22)

The method of claim 22, wherein the predetermined value is 40.

### Claim 24 (depends on 18)

The method of claim 18, wherein the number of outliers of step (f) is empirically chosen.

### Claim 25 (depends on 24)

The method of claim 24, wherein the number of outliers is five.

### Claim 26 (depends on 15)

The method of claim 15, wherein the identifying of outlying results, applying group delay algorithms, in audio files comprises the steps of:
a. generating alignments of the audio files at a phoneme level;
b. generating alignments of the audio files at a syllable level;
c. adjusting the alignments at the syllable level using group delay algorithms;
d. separating each syllable from the audio files into a separate audio file;
e. generating, for each separate audio file, phonemes of the separate audio files using phoneme boundaries for each syllable and an existing phoneme model;
f. determining a likelihood value of each generated phoneme, wherein if the likelihood value meets a criteria, identifying the generated phoneme as an outlier; and
g. identifying a sum of outliers for each sentence in the audio files, wherein if the sentence has more than a number of outliers, discarding the sentence from model training.

### Claim 27 (depends on 26)

The method of claim 26, wherein the generating of step (a) is performed using a speech recognition system.

### Claim 28 (depends on 26)

The method of claim 26, wherein the generating of step (b) is performed using at least one of: a speech recognition system and a phoneme model.

### Claim 29 (depends on 26)

The method of claim 26, wherein the criteria comprises a small value.

### Claim 30 (depends on 26)

The method of claim 26, wherein the criteria comprises a failed alignment.

### Claim 31 (depends on 26)

The method of claim 26, wherein the number of outliers of step (g) comprises three.
