Method to evaluate and fact-check an ai large language model chat response using domain- specific and domain-agnostic guidance for personalized medical provider-patient consult

Abstract

A method for evaluating an artificial intelligence (AI) large language model (LLM) generated response, the method comprising receiving a user query in the form of patient-related questions for a medical treatment domain; analyzing the user query using a LLM learned with open-source data and outputting a LLM answer from the LLM; performing a domain-specific evaluation of the LLM answer; performing a domain-agnostic evaluation of the LLM answer; generating at least one metric for the LLM answer based on the domain-specific evaluation and the domain-agnostic evaluation of the LLM answer; and evaluating the quality of the LLM answer based on the at least one metric.

Dependencies are inferred from the claim text.