Method and system for evaluating integration of responsible AI with LLM operations

A system for evaluating integration of Responsible Artificial Intelligence Operations (RAIOPS) and Large Language Model Operations (LLMOPS), the system comprising:

• at least one memory storing machine-executable instructions;
• and
• at least one processor communicatively coupled with the at least one memory, wherein the at least one processor executes the machine-executable instructions to perform operations comprising:
• generating, in response to receiving data associated with each prompt of a plurality of prompts, a response respective to each prompt of the plurality of prompts using at least one large language model (LLM);
• storing, in the at least one memory, the data associated with each prompt of the plurality of prompts and data associated with the response respective to each prompt as an association;
• generating, based at least in part upon a user-specified criteria and using the data associated with a subset of the plurality of prompts or the data associated with the response respective to the subset of the plurality of prompts, at least one evaluation metric for evaluating the response respective to each prompt of the plurality of prompts for at least one aspect of a plurality of aspects;
• generating, in accordance with the at least one evaluation metric, a knowledge graph visualization or a numerical score to display how the LLM is performing and for a user to determine whether the LLM needs optimization or tuning;
• and
• boosting the numerical score upon finding a synonym match in the response when compared with a respective ground-truth, and wherein the synonym match utilizes a Bidirectional Encoder Representations from Transformers (BERT) multilingual model,
• wherein the boosting comprises:
• calculating metrics specific to a domain or an application for the data associated with each prompt of the plurality of prompts and the data associated with the response respective to each prompt of the plurality of prompts,
• analyzing central tendencies of the metrics derived from multiple runs to generate different responses,
• setting a baseline score of each metric based on the analyzed central tendencies, and
• calculating a safe score based on the baseline score to determine the boosting of the numerical score.

A non-transitory computer-readable media (CRM) comprising instructions stored thereon for evaluating integration of Responsible Artificial Intelligence Operations (RAIOPS) and Large Language Model Operations (LLMOPS), wherein the instructions, when executed by at least one processor of a computing device, cause the computing device to perform operations comprising:

• generating, in response to receiving data associated with each prompt of a plurality of prompts, a response respective to each prompt of the plurality of prompts using at least one large language model (LLM);
• storing, in at least one memory, the data associated with each prompt of the plurality of prompts and data associated with the response respective to each prompt as an association;
• generating, based at least in part upon a user-specified criteria and using the data associated with a subset of the plurality of prompts or the data associated with the response respective to the subset of the plurality of prompts, at least one evaluation metric for evaluating the response respective to each prompt of the plurality of prompts for at least one aspect of a plurality of aspects;
• generating, in accordance with the at least one evaluation metric, a knowledge graph visualization or a numerical score to display how the LLM is performing and for a user to determine whether the LLM needs optimization or tuning;
• and
• boosting the numerical score upon finding a synonym match in the response when compared with a respective ground-truth, and wherein the synonym match utilizes a Bidirectional Encoder Representations from Transformers (BERT) multilingual model,
• wherein the boosting comprises:
• calculating metrics specific to a domain or an application for the data associated with each prompt of the plurality of prompts and the data associated with the response respective to each prompt of the plurality of prompts,
• analyzing central tendencies of the metrics derived from multiple runs to generate different responses,
• setting a baseline score of each metric based on the analyzed central tendencies, and
• calculating a safe score based on the baseline score to determine the boosting of the numerical score.