Evaluating conversation data based on risk factors
AssigneeNext IT Corporation
InventorsIan Beaver, Fred Brown, Casey Gossard
Life of this patent
Estimating the term…
Abstract
This disclosure describes techniques and architectures for evaluating conversations. In some instances, conversations with users, virtual assistants, and others may be analyzed to identify potential risks within a language model that is employed by the virtual assistants and other entities. The potential risks may be evaluated by administrators, users, systems, and others to identify potential issues with the language model that need to be addressed. This may allow the language model to be improved and enhance user experience with the virtual assistants and others that employ the language model.
Claims25
One or more non-transitory computer-readable media storing computer-readable instructions that, when executed, instruct one or more processors to perform acts comprising:
- causing a conversation user interface to be displayed on a display device to enable a conversation between a user and a virtual assistant;
- receiving, via the conversation user interface, user input;
- processing the user input with one or more natural language processing techniques to identify an intent unit for the user input, the intent unit being associated with a language model for the natural language processing techniques;
- determining, based at least in part on a presence of one or more risk indicators for the user input, a measure of confidence that the intent unit is correctly identified for the user input;
- associating the measure of confidence with the user input;
- determining a health status of the intent unit based at least in part on the measure of confidence, the health status indicating a level of risk associated with the intent unit;
- presenting the health status of the intent unit via an output device associated with an administrator;
- receiving a selection of the intent unit to obtain feedback regarding the intent unit;
- presenting a feedback interface that enables a voter to provide feedback regarding matching of the user input to the intent unit;
- receiving feedback for the voter regarding an accuracy of matching the user input to the intent unit;
- and
- evaluating the intent unit based at least in part on the feedback.
A method comprising:
- under control of a computing device configured with executable instructions, identifying one or more risk factors to evaluate conversation data associated with one or more users, the conversation data representing at least one conversation;
- processing the conversation data with a natural language processing system to identify an intent for the conversation data;
- determining a confidence value for the conversation data based at least in part on the one or more risk factors, the confidence value indicating a level of confidence that the intent is accurately identified for the conversation data;
- and
- utilizing the confidence value.
A system comprising:
- one or more processors;
- a database storing (i) conversation data representing at least one conversation and (ii) one or more risk factors;
- memory communicatively coupled to the one or more processors and storing executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising:
- processing the conversation data with a natural language processing system to identify an intent for the conversation data;
- determining a confidence value for the conversation data based at least in part on the one or more risk factors, the confidence value indicating a level of confidence that the intent is accurately identified for the conversation data;
- and
- utilizing the confidence value.
Dependencies are inferred from the claim text.