System, method, and computer program for evolving multi-turn chatbot dialogs

A method, comprising:

• at a computer system, evolving a large language model (LLM)-based chatbot over at least one iteration that includes:
• presenting, by a large language model (LLM)-based evaluator, a question to a LLM-based chatbot during a dialog with the LLM-based chatbot comprised of a sequence of question and answer pairs;
• receiving, by the LLM-based evaluator, an answer to the question from the LLM-based chatbot;
• evaluating, by the LLM-based evaluator, the answer according to one or more evaluation metrics and a ground truth;
• determining, by the LLM-based evaluator, that a result of the evaluation is unsatisfactory;
• and
• presenting, by the LLM-based evaluator, a follow-up question to the LLM-based chatbot designed to encourage a new answer of the LLM-based chatbot to be satisfactory with respect to the ground truth and to cause an optimization of the LLM-based chatbot.

A system, comprising:

• a non-transitory memory storing instructions;
• and
• one or more processors in communication with the non-transitory memory that execute the instructions to evolve a large language model (LLM)-based chatbot over at least one iteration that includes:
• presenting, by a large language model (LLM)-based evaluator, a question to a LLM-based chatbot during a dialog with the LLM-based chatbot comprised of a sequence of question and answer pairs;
• receiving, by the LLM-based evaluator, an answer to the question from the LLM-based chatbot;
• evaluating, by the LLM-based evaluator, the answer according to one or more evaluation metrics and a ground truth;
• determining, by the LLM-based evaluator, that a result of the evaluation is unsatisfactory;
• and
• presenting, by the LLM-based evaluator, a follow-up question to the LLM-based chatbot designed to encourage a new answer of the LLM-based chatbot to be satisfactory with respect to the ground truth and to cause an optimization of the LLM-based chatbot.