Dialog flow evaluation

An apparatus, comprising:

• at least one processor;
• and
• a computer readable storage medium having computer readable program code embodied therewith and executable by the at least one processor, the computer readable program code comprising:
• computer readable program code configured to receive information corresponding to a dialog flow between a virtual assistant and a user, wherein the information comprises dialog logs from previous conversations between the virtual assistant and users;
• computer readable program code configured to identify, from the received information, dialog states, wherein each dialog state corresponds to a possible response by the virtual assistant, wherein the identifying comprises analyzing the dialog logs and identifying responses provided by the virtual assistant;
• computer readable program code configured to classify a set of the identified dialog states as grounded dialog states, wherein the classifying comprises (i) receiving scoring input representing a desirability for each of the grounded dialog states and (ii) generating a score vector, for each of the grounded dialog states, based upon the received scoring input for each of the grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state, wherein the grounded dialog state comprises a dialog state having a scoring input provided by another user;
• computer readable program code configured to assign a score vector to each of the identified dialog states other than the grounded dialog states, wherein the assigning a score vector comprises extrapolating the score vector of a grounded dialog state to dialog states that the grounded dialog state is dependent upon;
• and
• computer readable program code configured to compute an aggregate score vector for the dialog flow by weighting the score vectors for each of the identified dialog states within the dialog flow and averaging the weighted dialog state score vectors, wherein the aggregate score vector identifies an overall desirability of the dialog flow.

A computer program product, comprising:

• a computer readable storage medium having computer readable program code embodied therewith, the computer readable program code executable by a processor and comprising:
• computer readable program code configured to receive information corresponding to a dialog flow between a virtual assistant and a user, wherein the information comprises dialog logs from previous conversations between the virtual assistant and users;
• computer readable program code configured to identify, from the received information, dialog states, wherein each dialog state corresponds to a possible response by the virtual assistant, wherein the identifying comprises analyzing the dialog logs and identifying responses provided by the virtual assistant;
• computer readable program code configured to classify a set of the identified dialog states as grounded dialog states, wherein the classifying comprises (i) receiving scoring input representing a desirability for each of the grounded dialog states and (ii) generating a score vector, for each of the grounded dialog states, based upon the received scoring input for each of the grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state, wherein the grounded dialog state comprises a dialog state having a scoring input provided by another user;
• computer readable program code configured to assign a score vector to each of the identified dialog states other than the grounded dialog states, wherein the assigning a score vector comprises extrapolating the score vector of a grounded dialog state to dialog states that the grounded dialog state is dependent upon;
• and
• computer readable program code configured to compute an aggregate score vector for the dialog flow by weighting the score vectors for each of the identified dialog states within the dialog flow and averaging the weighted dialog state score vectors, wherein the aggregate score vector identifies an overall desirability of the dialog flow.

A method, comprising:

• accessing at least one dialog model corresponding to a digital assistant, wherein the at least one dialog model identifies a dialog flow between the digital assistant and a user, wherein the at least one dialog model identifies a plurality of dialog nodes that correspond to possible responses that can be provided by the dialog assistant;
• identifying dialog states within the dialog model, wherein the dialog states correspond to responses provided by the digital assistant;
• receiving input, from a user, scoring each of at least a portion of the identified dialog states, wherein the score for each dialog state identifies a desirability of the dialog state;
• classifying the scored dialog states as grounded dialog states, wherein the classifying comprises classifying more than one of the dialog states as a grounded dialog state and wherein each of the grounded dialog states is separated by at least one other dialog state;
• scoring, based upon the score of the grounded dialog states, each of the remaining identified dialog states, wherein the scoring comprises statistically generating a score for each of the remaining identified dialog states based upon the dependence of the grounded dialog states to each of the remaining identified dialog states;
• and
• generating, using the grounded dialog states, an aggregate score for the at least one dialog model by weighting the score of each dialog state based upon a frequency of occurrence of the dialog state and averaging the weighted dialog state scores, wherein the aggregate score identifies an overall desirability of the dialog flow.