Continually evaluating and modifying artificial intelligence assistant

A system comprising:

• one or more memory devices;
• and
• one or more processors coupled to the one or more memory devices, the one or more processors configured to cause the system to:
• receive a prompt via an artificial intelligence assistant graphical user interface;
• generate, using a large language model based artificial intelligence assistant, a response to the prompt;
• determine an error in the response to the prompt by:
• generating, using an annotation tool, an annotated response by modifying one or more of the prompt or the response;
• providing the annotated response to one or more reviewer devices via an error graphical user interface;
• and
• receiving an indication of the error from the one or more reviewer devices provided via the error graphical user interface;
• classify the error as a high-severity error rather than a mid-severity error or a low-severity error by determining that the response includes a hallucination;
• and
• generate a modification to one or more components of the large language model based artificial intelligence assistant that addresses the high-severity error.

A computer-implemented method comprising:

• receiving, via one or more graphical user interfaces, a plurality of prompts;
• generating, using a large language model based artificial intelligence assistant, a plurality of responses to the plurality of prompts;
• performing a step for determining a plurality of errors in the plurality of prompts;
• performing a step for classifying the plurality of errors as one of high-severity, mid-severity, or low-severity;
• and
• generating a modification to the large language model based artificial intelligence assistant that addresses one or more errors classified as high-severity.