US 2026/0037877 A1 · EVALUATION AND FINE-TUNING OF GENERATIVE ARTIFICIAL INTELLIGENCE TOOLS

Abstract

Systems and methods of evaluating and fine-tuning a generative AI tool on a communication platform. The communication platform accesses a dataset comprising a user query and a response generated by an AI-based query system. The communication platform evaluates the response with respect to the user query using multiple AI-based scoring models to obtain multiple evaluation results. In response to determining that the multiple evaluation results are inconsistent, the communication platform evaluates the response with respect to the user query using a reference large language model (LLM) to provide a reference evaluation result. In response to determining that the reference evaluation result is decisive, the communication platform classifies, based on the reference evaluation result, the dataset to a data category of one or more data categories. The communication platform fine-tunes the AI-based query system based on a group of datasets in the data category.

Claims20

Dependent claims

A method comprising:

accessing a dataset comprising a user query and a response generated by an artificial intelligence (AI)-based query system;
evaluating the response with respect to the user query using multiple AI-based scoring models to obtain multiple evaluation results;
in response to determining that the multiple evaluation results are inconsistent, evaluating the response with respect to the user query using a reference large language model (LLM) to provide a reference evaluation result;
in response to determining that the reference evaluation result is decisive, classifying, based on the reference evaluation result, the dataset to a data category of one or more data categories;
and
fine-tuning the AI-based query system based on a group of datasets in the data category.

3 more elements

A system comprising:

a communications interface;
a non-transitory computer-readable medium;
and
one or more processors communicatively coupled to the communications interface and the non-transitory computer-readable medium, the one or more processors configured to execute processor-executable instructions stored in the non-transitory computer-readable medium to:
access a dataset comprising a user query and a response generated by an artificial intelligence (AI)-based query system;
evaluate the response with respect to the user query using multiple AI-based scoring models to obtain multiple evaluation results;
in response to determining that the multiple evaluation results are inconsistent, evaluate the response with respect to the user query using a reference large language model (LLM) to provide a reference evaluation result;
in response to determining that the reference evaluation result is decisive, classify, based on the reference evaluation result, the dataset to a data category of one or more data categories;
and
fine-tune the AI-based query system based on a group of datasets in the data category.

7 more elements

A non-transitory computer-readable medium comprising processor-executable instructions configured to cause one or more processors to:

access a dataset comprising a user query and a response generated by an artificial intelligence (AI)-based query system;
evaluate the response with respect to the user query using multiple AI-based scoring models to obtain multiple evaluation results;
in response to determining that the multiple evaluation results are inconsistent, evaluate the response with respect to the user query using a reference large language model (LLM) to provide a reference evaluation result;
in response to determining that the reference evaluation result is decisive, classify, based on the reference evaluation result, the dataset to a data category of one or more data categories;
and
fine-tune the AI-based query system based on a group of datasets in the data category.

3 more elements

Dependencies are inferred from the claim text.

Evaluation and fine-tuning of generative artificial intelligence tools

Life of this patent

Evaluation and fine-tuning of generative artificial intelligence tools

Life of this patent

Abstract

Claims20