# US 10,319,366 B2 · Predicting recognition quality of a phrase in automatic speech recognition systems

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 10,319,366 B2
- Document ID: US10319366B2
- Status: Granted
- Kind: B2
- Application number: 15/478,108
- Filed: 2017-04-03
- Published: 2017-07-20
- Granted: 2019-06-11
- Earliest claimed: 2013-10-30
- Applicants: Genesys Telecommunications Laboratories, Inc.
- Inventors: Amir Lev-Tov; Avraham Faizakof; Yochai Konig
- Web page: https://patentgrep.com/patent/US10319366B2

## CPC

- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L 15/02: Feature extraction for speech recognition; Selection of recognition unit
- G10L 15/04: Segmentation; Word boundary detection
- G10L 15/063: {Training}
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 25/30: using neural networks
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

A method for predicting a speech recognition quality of a phrase comprising at least one word includes: receiving, on a computer system including a processor and memory storing instructions, the phrase; computing, on the computer system, a set of features comprising one or more features corresponding to the phrase; providing the phrase to a prediction model on the computer system and receiving a predicted recognition quality value based on the set of features; and returning the predicted recognition quality value.

## Claims

### Claim 1

A method for configuring a speech analytics system, the method comprising:
computing, on a computer system comprising a processor and memory storing instructions, a plurality of text features corresponding to a text phrase, the text phrase comprising text including at least one word;
computing, by supplying the plurality of text features corresponding to the text phrase as input to a prediction model on the computer system, a predicted recognition quality value representing a likelihood of the text phrase being correctly recognized by an automatic speech recognition system of the speech analytics system when appearing in spoken form in user speech, the prediction model being trained using:
a plurality of true transcriptions of a collection of recorded speech;
a plurality of training text features of the true transcriptions; and
a recognizer output generated by supplying the recorded speech to the automatic speech recognition system; and
displaying, on a graphical user interface, the predicted recognition quality value for the text phrase.

### Claim 2 (depends on 1)

The method of claim 1, further comprising receiving the text of the phrase via the graphical user interface.

### Claim 3 (depends on 1)

The method of claim 1, further comprising displaying, on the graphical user interface, a label corresponding to the phrase, the label being based on the predicted recognition quality value.

### Claim 4 (depends on 1)

The method of claim 1, wherein the prediction model is a neural network.

### Claim 5 (depends on 4)

The method of claim 4, wherein the neural network is a multilayer perceptron neural network and wherein the neural network is trained by applying a backpropagation algorithm.

### Claim 6 (depends on 1)

The method of claim 1, wherein the prediction model is generated by:
generating, on the computer system, a plurality of training phrases from the collection of recorded speech;
calculating, on the computer system, a target value for each of the training phrases;
calculating the plurality of training text features of each of the training phrases;
training, on the computer system, the prediction model based on the training text features; and
setting, on the computer system, a filtering threshold.

### Claim 7 (depends on 6)

The method of claim 6, wherein the generating the training phrases comprises:
segmenting the plurality of true transcriptions into a plurality of true phrases;
processing the collection of recorded speech using the automatic speech recognition system to generate the recognizer output;
comparing the recognizer output to the true phrases to identify matches between the recognizer output and the true phrases;
based on the comparison, tagging matching phrases among the true phrases that match the recognizer output as hits;
selecting the matching phrases with a number of hits greater than a threshold value as training phrases; and
returning the plurality of training phrases.

### Claim 8 (depends on 6)

The method of claim 6, wherein the filtering threshold is set by optimizing precision and recall values on a test set of phrases of the plurality of training phrases.

### Claim 9 (depends on 1)

The method of claim 1, wherein the features of the phrase comprise at least one of:
a precision of a word in the phrase;
a recall of a word in the phrase;
a phrase error rate;
a sum of the precision of the phrase and the recall of the phrase;
a number of long words in the phrase;
a number of vowels in the phrase;
a length of the phrase;
a confusion matrix of the phrase; and
a feature of a language model.

### Claim 10 (depends on 1)

The method of claim 1, further comprising:
comparing the predicted recognition quality value to a threshold value; and
computing the likelihood of the phrase being correctly recognized by the automatic speech recognition system when appearing in audio based on the comparison between the predicted recognition quality value and the threshold value.

### Claim 11

A system for configuring a speech analytics system, the system comprising:
a processor; and
memory storing instructions that, when executed by the processor, cause the processor to:
compute a plurality of text features corresponding to a text phrase, the text phrase comprising text including at least one word;
compute, by supplying the plurality of text features corresponding to the text phrase as input to a prediction model, a predicted recognition quality value representing a likelihood of the text phrase being correctly recognized by an automatic speech recognition system of the speech analytics system when appearing in spoken from in user speech, the prediction model being trained using:
a plurality of true transcriptions of a collection of recorded speech;
a plurality of training text features of the true transcriptions; and
a recognizer output generated by supplying the recorded speech to the automatic speech recognition system; and
display, on a graphical user interface, the predicted recognition quality value for the text phrase.

### Claim 12 (depends on 11)

The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to receive the text of the phrase via the graphical user interface.

### Claim 13 (depends on 11)

The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to display, on the graphical user interface, a label corresponding to the phrase, the label being based on the predicted recognition quality value.

### Claim 14 (depends on 11)

The system of claim 11, wherein the prediction model is a neural network.

### Claim 15 (depends on 14)

The system of claim 14, wherein the neural network is a multilayer perceptron neural network and wherein the neural network is trained by applying a backpropagation algorithm.

### Claim 16 (depends on 11)

The system of claim 11, wherein the prediction model is generated by:
generating a plurality of training phrases from the collection of recorded speech;
calculating a target value for each of the training phrases;
calculating the plurality of training text features of each of the training phrases;
training the prediction model based on the training text features; and
setting a filtering threshold.

### Claim 17 (depends on 16)

The system of claim 16, wherein the generating the training phrases comprises:
segmenting the plurality of true transcriptions into a plurality of true phrases;
processing the collection of recorded speech using the automatic speech recognition system to generate the recognizer output;
comparing the recognizer output to the true phrases to identify matches between the recognizer output and the true phrases;
based on the comparison, tagging matching phrases among the true phrases that match the recognizer output as hits;
selecting the matching phrases with a number of hits greater than a threshold value as training phrases; and
returning the plurality of training phrases.

### Claim 18 (depends on 16)

The system of claim 16, wherein the filtering threshold is set by optimizing precision and recall values on a test set of phrases of the plurality of training phrases.

### Claim 19 (depends on 11)

The system of claim 11, wherein the features of the phrase comprise at least one of:
a precision of a word in the phrase;
a recall of a word in the phrase;
a phrase error rate;
a sum of the precision of the phrase and the recall of the phrase;
a number of long words in the phrase;
a number of vowels in the phrase;
a length of the phrase;
a confusion matrix of the phrase; and
a feature of a language model.

### Claim 20 (depends on 11)

The system of claim 11, wherein the memory further stores instructions that, when executed by the processor, cause the processor to:
compare the predicted recognition quality value to a threshold value; and
compute the likelihood of the phrase being correctly recognized by the automatic speech recognition system when appearing in audio based on the comparison between the predicted recognition quality value and the threshold value.