Content quality evaluation for retrieval augmented generation (rag) systems

A system comprising:

• an evaluation metric generator stored in memory and executable to:
• receive question-answer information for a data chunk residing in a source index, the question-answer information including at least a first test question answered by information in the data chunk;
• prompt a large language model (LLM) to generate one or more answer construct conditions for the first test question, each of the one or more answer construct conditions identifying a condition that is satisfied by a ground truth answer to the first test question;
• generate a question-specific evaluation metric for the first test question based on the one or more answer construct conditions;
• and
• a retrieval augmented generation (RAG) performance evaluator stored in memory and executable to:
• prompt multiple RAG systems to answer the first test question based on information within the source index, each of the multiple RAG systems being configured according to a different set of configurable parameters;
• quantifying quality of each of multiple responses to the first test question output by the multiple RAG systems by assessing the question-specific evaluation metric in association with each of the multiple answers;
• and
• present, on a user interface, comparative quality data quantifying the quality of the multiple responses generated by the multiple RAG systems relative to one another.

One or more tangible computer-readable storage media encoding computer-executable instructions for executing a computer process, the computer process comprising:

• prompting an LLM to generate question-answer pairs from data chunks in a source index, each of the question-answer pairs including a test question and a ground truth answer that are both derived from a select data chunk in the source index;
• prompting a large language model (LLM) to generate one or more answer construct conditions from the ground truth answer of each of the question-answer pairs, each of the one or more answer construct conditions identifying a condition that is satisfied by the corresponding ground truth answer;
• generating a question-specific evaluation metric for a first test question based on the one or more answer construct conditions derived from the ground truth answer to the first test question;
• conducting a response quality evaluation that entails:
• prompting multiple retrieval augmented generation (RAG) systems to answer the first test question based on information within the source index, each of the multiple RAG systems being configured according to a different set of user-configurable parameters;
• using the question-specific evaluation metric to quantify quality of each of multiple responses to the first test question output by the multiple RAG systems;
• and
• presenting, on a user interface, comparative quality data indicative of the quality of the multiple responses generated by the multiple RAG systems relative to one another.