Evaluation framework for intent authoring processes

A system for evaluating intent authoring processes, comprising:

• a processor executing instructions stored in a memory device;
• wherein the processor:
• receives a dataset comprising utterances of interactive dialog sessions between agents and clients for a given product or service;
• performs a classification of at least a portion of the utterances for a target intent according to at least one of a plurality of recommendation algorithms by an automatic driver;
• wherein the automatic driver invokes the at least one of the plurality of recommendation algorithms and simulates a manual confirmation of decisions generated by the one of the plurality of recommendation algorithms which would otherwise be performed by a user;
• automatically evaluates the classification of the utterances made by the automatic driver according to the at least one of the plurality of recommendation algorithms;
• and
• generates a report tracking the evaluated classification for each of the at least one of the plurality of recommendation algorithms.

A computer program product for evaluating intent authoring processes, the computer program product comprising a non-transitory computer-readable storage medium having computer-readable program code portions stored therein, the computer-readable program code portions comprising:

• an executable portion that receives a dataset comprising utterances of interactive dialog sessions between agents and clients for a given product or service;
• an executable portion that performs a classification of at least a portion of the utterances for a target intent according to at least one of a plurality of recommendation algorithms by an automatic driver;
• wherein the automatic driver invokes the at least one of the plurality of recommendation algorithms and simulates a manual confirmation of decisions generated by the one of the plurality of recommendation algorithms which would otherwise be performed by a user;
• an executable portion that automatically evaluates the classification of the utterances made by the automatic driver according to the at least one of the plurality of recommendation algorithms;
• and
• an executable portion that generates a report tracking the evaluated classification for each of the at least one of the plurality of recommendation algorithms.