Device and method for evaluating speech recognition system
AssigneesHyundai Motor Company, Kia Corporation
InventorChang Woo Chun
Life of this patent
Estimating the term…
Abstract
An embodiment device for evaluating a speech recognition system including a plurality of speech recognition engines and a plurality of natural language understanding (NLU) engines includes one or more processors and at least one storage device storing a program to be executed by the one or more processors, the program including instructions to obtain speech recognition results in which the plurality of speech recognition engines recognize input audio, evaluate the plurality of speech recognition engines based on a comparison between the speech recognition results, obtain NLU results in which the plurality of NLU engines understand each speech recognition result, and evaluate the plurality of NLU engines based on a comparison between the NLU results.
Claims20
A device for evaluating a speech recognition system including a plurality of speech recognition engines and a plurality of natural language understanding (NLU) engines, the device comprising:
- one or more processors;
- and
- at least one storage device storing a program to be executed by the one or more processors, the program including instructions to:
- obtain speech recognition results in which the plurality of speech recognition engines recognize input audio;
- evaluate the plurality of speech recognition engines based on a comparison between the speech recognition results;
- obtain NLU results in which the plurality of NLU engines understand each speech recognition result;
- evaluate the plurality of NLU engines based on a comparison between the NLU results;
- detect recognition failure of the input audio by comparing NLU results of one NLU engine understanding the speech recognition results with an NLU label for the speech recognition results;
- and
- in response to a determination indicating that the NLU results of the one NLU engine are the same as the NLU label for the speech recognition results, but at least two of the speech recognition results are different, determine that the NLU results of the one NLU engine contain a defect.
A computer implemented method for evaluating a speech recognition system including a plurality of speech recognition engines and a plurality of natural language understanding (NLU) engines, the method comprising:
- obtaining speech recognition results in which the plurality of speech recognition engines recognize input audio;
- obtaining NLU results in which the plurality of NLU engines understand each speech recognition result;
- evaluating the plurality of speech recognition engines based on a comparison between the speech recognition results;
- evaluating the plurality of NLU engines based on a comparison between the NLU results;
- detecting recognition failure of the input audio by comparing NLU results of one NLU engine understanding the speech recognition results with an NLU label for the speech recognition results;
- and
- determining that the NLU results of the one NLU engine contain a defect in a case in which the NLU results of the one NLU engine are the same as the NLU label for the speech recognition results, but at least two of the speech recognition results are different.
A non-transitory computer-readable storage medium storing instructions that, when executed by one or more computers, cause the one or more computers to perform operations, the operations comprising:
- recognizing input audio;
- obtaining speech recognition results generated by each of a plurality of speech recognition engines based on the recognized input audio;
- obtaining natural language understanding (NLU) results generated by each of a plurality of NLU engines based on the obtained speech recognition results;
- evaluating the plurality of speech recognition engines based on a comparison between the speech recognition results;
- evaluating the plurality of NLU engines based on a comparison between the NLU results;
- detecting recognition failure of the input audio by comparing (i) particular NLU results of a particular NLU engine, of the plurality of NLU engines, and (ii) an NLU label for the speech recognition results;
- determining that the particular NLU results of the particular NLU engine (i) are the same as the NLU label for the speech recognition results and (ii) that at least two of the speech recognition results are different;
- and
- in response to the determination that the NLU results of the particular NLU engine (i) are the same as the NLU label for the speech recognition results and (ii) that at least two of the speech recognition results are different, determining that the NLU results of the particular NLU engine contain a defect.
Dependencies are inferred from the claim text.