# US 2020/0286476 A1 · Characterizing accuracy of ensemble models for automatic speech recognition

> Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt

- Number: US 2020/0286476 A1
- Document ID: US20200286476A1
- Status: Published application
- Kind: A1
- Application number: 16/297,602
- Filed: 2019-03-09
- Published: 2020-09-10
- Earliest claimed: 2019-03-09
- Assignees: RIZIO, Inc.
- Applicants: Rizio LLC
- Inventors: Ahmad ABDULKADER; Mohamed Gamal Mohamed MAHMOUD
- Web page: https://patentgrep.com/patent/US20200286476A1

## CPC

- G10L 15/22: Procedures used during a speech recognition process, e.g. man-machine dialogue
- G10L 15/01: Assessment or evaluation of speech recognition systems
- G10L 15/063: {Training}
- G10L 15/16: using artificial neural networks
- G10L 15/26: Speech to text systems (G10L15/08 takes precedence)
- G10L 15/32: Multiple recognisers used in sequence or in parallel; Score combination systems therefor, e.g. voting systems
- G10L 2015/0635: {updating or merging of old and new templates; Mean values; Weighting}
- G10L: SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING

## Abstract

One embodiment of the present invention sets forth a technique for analyzing a transcription of a recording. The technique includes generating features representing transcriptions produced by multiple automatic speech recognition (ASR) engines from voice activity in the recording and a best transcription of the recording produced by an ensemble model from the transcriptions. The technique also includes applying a machine learning model to the features to produce a score representing an accuracy of the best transcription. The technique further includes storing the score in association with the best transcription.

## Claims

### Claim 1

A method for analyzing a transcription of a recording, comprising:
generating features representing transcriptions produced by multiple automatic speech recognition (ASR) engines from voice activity in the recording and a best transcription of the recording produced by an ensemble model from the transcriptions;
applying a machine learning model to the features to produce a score representing an accuracy of the best transcription; and
storing the score in association with the best transcription.

### Claim 2 (depends on 1)

The method of claim 1, further comprising:
applying one or more thresholds to the score to characterize the accuracy of the best transcription; and
determining a candidacy of the recording for human transcription based on the characterized accuracy of the best transcription.

### Claim 3 (depends on 2)

The method of claim 2, further comprising:
generating training data for the ensemble model from the best transcription and the human transcription; and
updating parameters of the ensemble model based on the training data.

### Claim 4 (depends on 2)

The method of claim 2, wherein determining the candidacy of the recording for human transcription based on the characterized accuracy of the best transcription comprises:
identifying the recording as a candidate for human transcription when the score falls between a first threshold for a high error rate and a second threshold for a low error rate.

### Claim 5 (depends on 1)

The method of claim 1, wherein generating the features representing the transcriptions produced by the multiple ASR engines from voice activity in the recording and the best transcription of the recording produced by the ensemble model from the transcriptions comprises:
generating a first set of features from the transcriptions;
generating a second set of features from pairwise comparisons of the transcriptions;
generating a third set of features from the best transcription; and
generating a fourth set of features from the recording.

### Claim 6 (depends on 5)

The method of claim 5, wherein the first set of features comprises a length of a transcription, a confidence in the transcription, and a letters per second associated with the transcription.

### Claim 7 (depends on 5)

The method of claim 5, wherein the second set of features comprises a word error rate between two transcriptions, an average word error rate across all pairs of transcriptions, an difference in length between the two transcriptions, and an average difference in length across all pairs of transcriptions.

### Claim 8 (depends on 5)

The method of claim 5, wherein the third set of features comprises a first feature representing a pairwise comparison of the best transcription and each of the transcriptions and a second feature representing an attribute of the best transcription.

### Claim 9 (depends on 5)

The method of claim 5, wherein the fourth set of features comprises a duration of the voice activity, a position of the voice activity in the recording, and an audio feature.

### Claim 10 (depends on 9)

The method of claim 9, wherein the audio feature comprises at least one of a mel-frequency cepstral coefficient (MFCC), a perceptual linear prediction (PLP), a root mean square (RMS), a zero crossing rate, a spectral flux, a spectral energy, a chroma vector, and a chroma deviation.

### Claim 11 (depends on 5)

The method of claim 5, wherein the second set of features comprises a fixed-size encoding of per-character differences between two transcriptions.

### Claim 12 (depends on 1)

The method of claim 1, wherein the machine learning model comprises an artificial neural network.

### Claim 13

A non-transitory computer readable medium storing instructions that, when executed by a processor, cause the processor to perform the steps of:
generating features representing transcriptions produced by multiple automatic speech recognition (ASR) engines from voice activity in the recording and a best transcription of the recording produced by an ensemble model from the transcriptions;
applying a machine learning model to the features to produce a score representing an accuracy of the best transcription; and
storing the score in association with the best transcription.

### Claim 14 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein the steps further comprise:
applying one or more thresholds to the score to characterize the accuracy of the best transcription; and
determining a candidacy of the recording for human transcription based on the characterized accuracy of the best transcription.

### Claim 15 (depends on 14)

The non-transitory computer readable medium of claim 14, wherein the steps further comprise:
generating training data for the ensemble model from the best transcription and the human transcription; and
updating parameters of the ensemble model based on the training data.

### Claim 16 (depends on 13)

The non-transitory computer readable medium of claim 13, wherein generating the features representing the transcriptions produced by the multiple ASR engines from voice activity in the recording and the best transcription of the recording produced by the ensemble model from the transcriptions comprises:
generating a first set of features from the transcriptions;
generating a second set of features from pairwise comparisons of the transcriptions;
generating a third set of features from the best transcription; and
generating a fourth set of features from the recording.

### Claim 17 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the first set of features and third set of features comprise a length of a transcription, a confidence in the transcription, and a letters per second associated with the transcription

### Claim 18 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the second set of features and third set of features comprises a word error rate between two transcriptions, an average word error rate across all pairs of transcriptions, an difference in length between the two transcriptions, and an average difference in length across all pairs of transcriptions.

### Claim 19 (depends on 16)

The non-transitory computer readable medium of claim 16, wherein the fourth set of features comprises a duration of the voice activity, a position of the voice activity in the recording, and an audio feature.

### Claim 20

A system, comprising:
a memory that stores instructions; and
a processor that is coupled to the memory and, when executing the instructions, is configured to:
generate features representing transcriptions produced by multiple automatic speech recognition (ASR) engines from voice activity in the recording and a best transcription of the recording produced by an ensemble model from the transcriptions;
apply a machine learning model to the features to produce a score representing an accuracy of the best transcription; and
store the score in association with the best transcription.
