Detecting customers with low speech recognition accuracy by investigating consistency of conversation in call-center

A non-transitory computer readable storage medium comprising a computer readable program for estimating automatic speech recognition (ASR) accuracy, wherein the computer readable program when executed on a computer causes the computer to perform the steps of:

• obtaining transcriptions of utterances in a conversation over two channels;
• sorting the transcriptions along a time axis using a forced alignment;
• training a language model with the sorted transcriptions;
• performing ASR for utterances in a conversation between a first user and a second user, the second user being a target of ASR accuracy estimation;
• determining whether an ASR result of the second user is consistent or inconsistent with an ASR result of the first user using the trained language model;
• listing word sequences that have at least one word from the first user followed by a word from the second user, and wherein said determining step determines whether the ASR results of the first and second users are consistent or inconsistent by comparing an average of probabilities of the word from the second user across each of the word sequences against a threshold;
• and
• estimating the ASR result of the second user as poor responsive to the ASR result of the second user being as inconsistent with the ASR result of the first user.

A system for estimating automatic speech recognition (ASR) accuracy, the system comprising:

• a processor, configured to:
• obtain transcriptions of utterances in a conversation over two channels;
• sort the transcriptions along a time axis using a forced alignment;
• train a language model with the sorted transcriptions;
• perform ASR for utterances in a conversation between a first user and a second user, the second user being a target of ASR accuracy estimation;
• determine whether an ASR result of the second user is consistent or inconsistent with an ASR result of the first user using the trained language model;
• list word sequences that have at least one word from the first user followed by a word from the second user, and wherein said processor step determines whether the ASR results of the first and second users are consistent or inconsistent by comparing an average of probabilities of the word from the second user across each of the word sequences against a threshold;
• and
• estimate the ASR result of the second user as poor responsive to the ASR result of the second user being as inconsistent with the ASR result of the first user.

A method for estimating automatic speech recognition (ASR) accuracy, the method comprising:

• obtaining transcriptions of utterances in a conversation over two channels;
• sorting the transcriptions along a time axis using a forced alignment;
• training a language model with the sorted transcriptions;
• performing ASR for utterances in a conversation between a first user and a second user, the second user being a target of ASR accuracy estimation;
• determining whether an ASR result of the second user is consistent or inconsistent with an ASR result of the first user using the trained language model;
• listing word sequences that have at least one word from the first user followed by a word from the second user, and wherein said determining step determines whether the ASR results of the first and second users are consistent or inconsistent by comparing an average of probabilities of the word from the second user across each of the word sequences against a threshold;
• retraining the language model with any of the transcriptions from the second user being removed for the retraining to improve an accuracy of the language model, responsive to the ASR result of the second user being as inconsistent with the ASR result of the first user;
• and
• decoding a new utterance using the retrained language model.