Enterprise retrieval-augmented generation evaluation framework

A computer-implemented method, comprising:

• iteratively performing the following process, by a computer processor of an enterprise Retrieval-Augmented Generation (“RAG”) evaluation system coupled to an enterprise RAG system associated with at least one Large Language Model (“LLM”), until an evaluation threshold is satisfied:
• determining RAG generation metric scores including:
• (i) a faithfulness score measuring how factually consistent a generated answer is with the context information, and (ii) an answer relevance score measuring how correctly the generated answer addresses the question,
• determining RAG retrieval metric scores including (i) a context precision score measuring if all relevant items from the context information are retrieved, (ii) a context relevancy score measuring a relevancy of the retrieved context information to the question, and (iii) a context recall score measuring how well the retrieved context information matches the generated answer,
• determining end-to-end RAG metric scores including:
• (i) an answer semantic similarity score measuring how closely semantic content of the generated answer aligns with the associated ground truth, and (ii) an answer correctness score measuring a factual overlap between the generated answer and the ground truth answer, and
• automatically adjusting at least one RAG turning parameter;
• and
• when the evaluation threshold is satisfied, outputting a metrics report that includes the RAG generation metrics scores, the RAG retrieval metrics scores, and end-to-end RAG metrics scores.

One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by a computing system, cause the computing system to perform operations, comprising:

• iteratively performing, by a computer processor of an enterprise Retrieval-Augmented Generation (“RAG”) evaluation system coupled to an enterprise RAG system associated with at least one Large Language Model (“LLM”), the following process until an evaluation threshold is satisfied:
• determining an RAG generation metric score,
• determining an RAG retrieval metric score,
• determining an end-to-end RAG metric score, and
• automatically adjusting at least one RAG turning parameter;
• and
• when the evaluation threshold is satisfied, outputting a metrics report that includes the RAG generation metrics scores, the RAG retrieval metrics scores, and end-to-end RAG metrics scores.