Model-agnostic visualizations using linear programming approximation
AssigneeVerint Americas Inc.
InventorsAbhinav Aggarwal, Ian Roy Beaver, Cynthia Freeman, Jonathan Patrick Merriman
Life of this patent
Estimating the term…
Abstract
A method of determining influence of language elements in script to an overall classification of the script by perturbing the dataset representing a conversation. In some instances, for example, in the conversation, the language elements and turns within the conversation (e.g., in a chat bot) are analyzed for their influence in escalation or non-escalation of the conversation to a higher level of resolution, e.g., to a human representative or manager.
Claims14
A method of visually presenting a representation of the effect of language elements on outcome of a conversation C between a user and a virtual assistant to a representative, the method comprising:
- generating a graphical user interface and displaying a conversation C on the graphical user interface to the representative;
- identifying a set of turns T1, . . . , TN corresponding to the conversation C;
- creating a set of perturbations to be applied to data representing a set of textual language elements in conversation C between a user and a virtual assistant, wherein the virtual assistant uses a large language model;
- applying the set of perturbations to the data;
- determining a weight wi>0 for each of the turns Ti in the set of turns such that, given any perturbation P from the set of perturbations, a linear combination of the weights of the turns affected by perturbation P approximates a probability of escalation;
- formulating a plurality of linear programming approximations using results from the perturbations and solving the plurality of linear programming approximations to generate solutions;
- displaying on the graphical user interface a visual presentation to the representative based on the solutions and the probability of escalation, whereby visual aspects of the visual presentation represent relative possibility that one of the perturbations will change a classification of the conversation, the possibility based on the solutions, wherein the large language model was trained using one or more previous perturbations and solutions.
One or more non-transitory computer-readable storage media storing computer-readable instructions that, when execute, instruct one or more processors to perform operations, comprising:
- generating a graphical user interface and displaying a conversation C on the graphical user interface to a representative;
- identifying a set of turns T1, . . . , TN corresponding to the conversation C;
- creating a set of perturbations to be applied to data representing a set of textual language elements in conversation C between a user and a virtual assistant, wherein the virtual assistant uses a large language model;
- applying the set of perturbations to the data;
- determining a weight wi>0 for each of the turns Ti in the set of turns such that, given any perturbation P from the set of perturbations, a linear combination of the weights of the turns affected by perturbation P approximates a probability of escalation;
- formulating a plurality of linear programming approximation using results from the perturbations and solving the plurality of linear programming approximations to generate solutions;
- and
- displaying on the graphical user interface a visual presentation to a representative based on the solutions and the probability of escalation, whereby visual aspects of the visual presentation represent relative possibility that one of the perturbations will change a classification of the conversation, the possibility based on the solutions, wherein the large language model was trained using one or more previous perturbations and solutions.
A method of identifying effect of language elements on an outcome of a conversation between a user and a virtual assistant, the method comprising, comprising:
- generating a graphical user interface and displaying a conversation C on the graphical user interface to the user;
- for each turn Ti of the conversation C=(T1, . . . , TN) between the user and the virtual assistant, wherein the virtual assistant uses a large language model:
- removing data representing the turn Ti from a conversation dataset representing the conversation to produce a perturbed dataset, the conversation having a plurality of dialogue turns and the conversation having a classification of escalation or non-escalation before removing the data;
- applying a classifier to the perturbed dataset;
- and
- upon determining that the perturbed dataset has a different classification than the conversation dataset, displaying on the graphical interface a visual indication that the turn Ti represented by the removed data is important to an outcome of applying the classifier, wherein the large language model was trained using one or more previous perturbations and solutions.
A non-transitory computer-readable storage media having stored therein software executable to perform steps, comprising:
- generating a graphical user interface and displaying a conversation C between a user and a virtual assistant on the graphical user interface to the user, wherein the virtual assistant uses a large language model;
- for each turn Ti of a conversation C=(T1, . . . , TN) between the user and the virtual assistant:
- removing data representing the turn Ti from a conversation dataset representing the conversation to produce a perturbed dataset, the conversation having a plurality of dialogue turns and the conversation having a classification of escalation or non-escalation before removing the data;
- applying a classifier to the perturbed dataset;
- and
- upon determining that the perturbed dataset has a different classification than the conversation dataset, displaying on the graphical interface a visual indication that the turn Ti represented by the removed data is important to an outcome of applying the classifier, wherein the large language model was trained using one or more previous perturbations and solutions.
Dependencies are inferred from the claim text.