Systems and methods for an end-to-end evaluation and testing framework for task-oriented dialog systems

A system for dialog simulation, the system comprising:

• a communication interface that receives a plurality of task-oriented dialog data generated from a dialog agent;
• and
• one or more hardware processors that:
• determines a plurality of natural language understanding pairs including bot dialog acts and respective bot messages based on the plurality of task-oriented dialog data;
• determines a plurality of goal pairs including goal entity slots and respective goal entity slot values based on the plurality of task-oriented dialog data;
• determines a plurality of natural language generation templates based on the plurality of task-oriented dialog data;
• generates a simulated task-oriented dialog based on the plurality of natural language understanding pairs, the plurality of goal pairs, and the plurality of natural language generation templates;
• generates simulation results from the simulated task-oriented dialog;
• and
• generates, based on simulation results, an actionable suggestion relating to an adjustment to the dialog agent.

A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions, the instructions being executed by a processor to perform operations comprising:

• receiving, via a communication interface, a plurality of task-oriented dialog data generated from a dialog agent;
• determining a plurality of natural language understanding pairs including bot dialog acts and respective bot messages based on the plurality of task-oriented dialog data;
• determining a plurality of goal pairs including goal entity slots and respective goal entity slot values based on the plurality of task-oriented dialog data;
• determining a plurality of natural language generation templates based on the plurality of task-oriented dialog data;
• generating a simulated task-oriented dialog based on the plurality of natural language understanding pairs, the plurality of goal pairs, and the plurality of natural language generation templates;
• generating simulation results from the simulated task-oriented dialog;
• and
• generating, based on simulation results, an actionable suggestion relating to an adjustment to the dialog agent.