# US 12,112,138 B2 · Systems and methods for an end-to-end evaluation and testing framework for task-oriented dialog systems > Agents: this data is available as JSON from the patentgrep API and MCP server — see https://patentgrep.com/llms.txt - Number: US 12,112,138 B2 - Document ID: US12112138B2 - Status: Granted - Kind: B2 - Application number: 17/830,889 - Filed: 2022-06-02 - Published: 2023-07-27 - Granted: 2024-10-08 - Earliest claimed: 2022-01-27 - Assignees: Salesforce, Inc. - Applicants: Salesforce, Inc. - Inventors: Guangsen Wang; Samson Min Rong Tan; Shafiq Rayhan Joty; Gang Wu; Chu Hong Hoi; Ka Chun Au - Web page: https://patentgrep.com/patent/US12112138B2 ## CPC - G06F 40/35: Discourse or dialogue representation - G06F 40/186: Templates - G06F 40/30: Semantic analysis - G06F 40/40: Processing or translation of natural language (natural language analysis G06F40/20; semantic analysis G06F40/30) - G06F 40/56: Natural language generation - H04L 51/02: using automatic reactions or user delegation, e.g. automatic replies or chatbot-generated messages - G06F: ELECTRIC DIGITAL DATA PROCESSING (computer systems based on specific computational models G06N) - H04L: TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION (arrangements common to telegraphic and telephonic communication H04M) ## Abstract Embodiments provide a software framework for evaluating and troubleshooting real-world task-oriented bot systems. Specifically, the evaluation framework includes a generator that infers dialog acts and entities from bot definitions and generates test cases for the system via model-based paraphrasing. The framework may also include a simulator for task-oriented dialog user simulation that supports both regression testing and end-to-end evaluation. The framework may also include a remediator to analyze and visualize the simulation results, remedy some of the identified issues, and provide actionable suggestions for improving the task-oriented dialog system. ## Claims ### Claim 1 A method of simulating a dialog, the method comprising: receiving, via a communication interface, a plurality of task-oriented dialog data generated from a dialog agent; determining a plurality of natural language understanding pairs including bot dialog acts and respective bot messages based on the plurality of task-oriented dialog data; determining a plurality of goal pairs including goal entity slots and respective goal entity slot values based on the plurality of task-oriented dialog data; determining a plurality of natural language generation templates based on the plurality of task-oriented dialog data; generating a simulated task-oriented dialog based on the plurality of natural language understanding pairs, the plurality of goal pairs, and the plurality of natural language generation templates; generating simulation results from the simulated task-oriented dialog; and generating, based on simulation results, an actionable suggestion relating to an adjustment to the dialog agent. ### Claim 2 (depends on 1) The method of claim 1, wherein the plurality of task-oriented dialog data comprises responses to application programming interface (API) calls to the dialog agent. ### Claim 3 (depends on 1) The method of claim 1, wherein the plurality of task-oriented dialog data comprises metadata associated with the dialog agent. ### Claim 4 (depends on 1) The method of claim 1, further comprising: determining an intent training utterance based on the plurality of task-oriented dialog data. ### Claim 5 (depends on 4) The method of claim 4, further comprising: generating, by a plurality of models, a plurality of paraphrases based on the intent training utterance. ### Claim 6 (depends on 5) The method of claim 5, further comprising: filtering the plurality of paraphrases based on a similarity metric to produce a subset of the plurality of paraphrases. ### Claim 7 (depends on 6) The method of claim 6, further comprising: generating the simulated task-oriented dialog based on the subset of the plurality of paraphrases. ### Claim 8 A system for dialog simulation, the system comprising: a communication interface that receives a plurality of task-oriented dialog data generated from a dialog agent; and one or more hardware processors that: determines a plurality of natural language understanding pairs including bot dialog acts and respective bot messages based on the plurality of task-oriented dialog data; determines a plurality of goal pairs including goal entity slots and respective goal entity slot values based on the plurality of task-oriented dialog data; determines a plurality of natural language generation templates based on the plurality of task-oriented dialog data; generates a simulated task-oriented dialog based on the plurality of natural language understanding pairs, the plurality of goal pairs, and the plurality of natural language generation templates; generates simulation results from the simulated task-oriented dialog; and generates, based on simulation results, an actionable suggestion relating to an adjustment to the dialog agent. ### Claim 9 (depends on 8) The system of claim 8, wherein the plurality of task-oriented dialog data comprises responses to application programming interface (API) calls to the dialog agent. ### Claim 10 (depends on 8) The system of claim 8, wherein the plurality of task-oriented dialog data comprises metadata associated with the dialog agent. ### Claim 11 (depends on 8) The system of claim 8, wherein the one or more hardware processors further: determines an intent training utterance based on the plurality of task-oriented dialog data. ### Claim 12 (depends on 11) The system of claim 11, wherein the one or more hardware processors further: generates, by a plurality of models, a plurality of paraphrases based on the intent training utterance. ### Claim 13 (depends on 12) The system of claim 12, wherein the one or more hardware processors further: filters the plurality of paraphrases based on a similarity metric to produce a subset of the plurality of paraphrases. ### Claim 14 (depends on 13) The system of claim 13, wherein the one or more hardware processors further: generates the simulated task-oriented dialog based on the subset of the plurality of paraphrases. ### Claim 15 A processor-readable non-transitory storage medium storing a plurality of processor-executable instructions, the instructions being executed by a processor to perform operations comprising: receiving, via a communication interface, a plurality of task-oriented dialog data generated from a dialog agent; determining a plurality of natural language understanding pairs including bot dialog acts and respective bot messages based on the plurality of task-oriented dialog data; determining a plurality of goal pairs including goal entity slots and respective goal entity slot values based on the plurality of task-oriented dialog data; determining a plurality of natural language generation templates based on the plurality of task-oriented dialog data; generating a simulated task-oriented dialog based on the plurality of natural language understanding pairs, the plurality of goal pairs, and the plurality of natural language generation templates; generating simulation results from the simulated task-oriented dialog; and generating, based on simulation results, an actionable suggestion relating to an adjustment to the dialog agent. ### Claim 16 (depends on 15) The processor-readable non-transitory storage medium of claim 15, wherein the plurality of task-oriented dialog data comprises metadata associated with the dialog agent. ### Claim 17 (depends on 15) The processor-readable non-transitory storage medium of claim 15, further comprising: determining an intent training utterance based on the plurality of task-oriented dialog data. ### Claim 18 (depends on 17) The processor-readable non-transitory storage medium of claim 17, further comprising: generating, by a plurality of models, a plurality of paraphrases based on the intent training utterance. ### Claim 19 (depends on 18) The processor-readable non-transitory storage medium of claim 18, further comprising: filtering the plurality of paraphrases based on a similarity metric to produce a subset of the plurality of paraphrases. ### Claim 20 (depends on 19) The processor-readable non-transitory storage medium of claim 19, further comprising: generating the simulated task-oriented dialog based on the subset of the plurality of paraphrases.