Dynamic context-based evaluation of language models

An apparatus comprising:

• at least one processing device comprising a processor coupled to a memory;
• the at least one processing device being configured to implement the following steps:
• accessing at least one data structure comprising information characterizing at least one input from a user and a corresponding context of the at least one input;
• dynamically identifying, in response to the at least one input from the user, one or more evaluation criteria, from a plurality of evaluation criteria, for evaluating at least one response to be generated by at least one language model, based at least in part on the context of the at least one input;
• dynamically generating one or more evaluation metrics based at least in part on the one or more identified evaluation criteria;
• applying at least a portion of the input from the user to the at least one language model, wherein the at least one language model generates the at least one response based at least in part on the input from the user;
• evaluating the at least one response from the at least one language model using the dynamically generated one or more evaluation metrics to assign a score, using at least one processing device comprising a processor coupled to a memory, to one or more of the at least one response;
• and
• initiating at least one automated action based at least in part on the score assigned to the one or more responses, wherein the at least one automated action comprises one or more of (i) updating one or more of the at least one language model, at least one prompt applied to the at least one language model and at least one query applied to the at least one language model and (ii) retraining the at least one language model.

A non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes the at least one processing device to perform the following steps:

• accessing at least one data structure comprising information characterizing at least one input from a user and a corresponding context of the at least one input;
• dynamically identifying, in response to the at least one input from the user, one or more evaluation criteria, from a plurality of evaluation criteria, for evaluating at least one response to be generated by at least one language model, based at least in part on the context of the at least one input;
• dynamically generating one or more evaluation metrics based at least in part on the one or more identified evaluation criteria;
• applying at least a portion of the input from the user to the at least one language model, wherein the at least one language model generates the at least one response based at least in part on the input from the user;
• evaluating the at least one response from the at least one language model using the dynamically generated one or more evaluation metrics to assign a score, using at least one processing device comprising a processor coupled to a memory, to one or more of the at least one response;
• and
• initiating at least one automated action based at least in part on the score assigned to the one or more responses, wherein the at least one automated action comprises one or more of (i) updating one or more of the at least one language model, at least one prompt applied to the at least one language model and at least one query applied to the at least one language model and (ii) retraining the at least one language model.